knitr::opts_chunk$set(warning=FALSE)

Needed libraries

library(dplyr)
library(countrycode)
library(outliers)
library(caret)
library(cluster)
library(factoextra)
library(NbClust)
library("DMwR")
library("RWeka")
library("C50")
library("rpart")
library("themis")
library(rattle)
library(rpart.plot)
library(RColorBrewer)

phase 1

Problem statement

Prediction of cyber security employees’ salaries based on 11 attributes & grouping employees based on shared characteristics.

1.work_year

2.experience_level

3.employment_type

4.job_title

5.salary

6.salary_currency

7.salary_in_usd

8.employee_residence

9.remote_ratio

10.company_location

11.company_size

Problem description

We are living in the “information age” or rather the “data age”, meaning that everything around us revolves around data. The data has become one of the most valuable assets that a person or an organisation can have, since it has a significant value, losing it will lead to significant damages. Thus, most of the attacks nowadays are directed toward the data. To guard against such damages, organisations have realised the importance of protecting their digital assets, leading them to hire cybersecurity specialists. This made cybersecurity gain popularity among people so there’s a growing tendency to study cybersecurity. Consequently this resulted in the emergence of plentiful professionals with various experience levels and skills in this field. As a result, organisations may find it difficult to decide a salary for job candidates solely based on the CV. also, since the attacks improve rapidly, organisations need to hire more employees in the far future to defend against such attacks but it’s not an easy matter to predict the future payroll which may hinders some of the organisation’s plans. Another issue arises when the decision makers in the organisation aren’t fully aware of the different groups of employees and their differint needs. Their lack of awareness gives a chance for the competitor organisations to attract their employees to them by offering a better salary and privilages that match their needs.

Data mining task

Prediction of the cyber security employees’ salary categories (Very Low, Low, , High, Very High) using classification, and description of data characteristics and behavior and grouping data using clustering methods.

Goal

Given the problems we discussed and In order to better understand this field, we decided to analyse a dataset of 1247 cybersecurity employees, containing information such as salary, job title, and experience level. Analysing this dataset can provide insightful predictions regarding the salary range of a cybersecurity employee and description of the cybersecurity market behavior by grouping the data, which can help in:

  • Market segmentation
  • Identify trends
  • Specifying common charactrestics among cybersecurity employees
  • Identify the main cybersecurity employee groups for better understanding their needs
  • Making better decisions
  • Making recruitment and hiring process easier and more efficient
  • Predicting the future payroll
  • Increasing loyalty
  • Increasing the satisfaction rate
  • Achieving fairness

Source of data:

https://www.kaggle.com/datasets/deepcontractor/cyber-security-salaries

Reading and viewing dataset

dataset= read.csv(url("https://raw.githubusercontent.com/SarahAlhindi/DM_project/main/Data%20Set/salaries_cyber.csv"), header=TRUE)
View(dataset)

Original dataset

we will keep a copy of the original dataset before data preprocessing to use if needed at any time

originalDataset= dataset

General information about the dataset:

No. of attributes: 11
Type of attributes: Ordinal , Nominal, and Numeric
No. of objects: 1247
Class label: salary_in_usd

ncol(dataset)
[1] 11
nrow(dataset)
[1] 1247
names(dataset)
 [1] "work_year"          "experience_level"   "employment_type"   
 [4] "job_title"          "salary"             "salary_currency"   
 [7] "salary_in_usd"      "employee_residence" "remote_ratio"      
[10] "company_location"   "company_size"      
str(dataset)
'data.frame':   1247 obs. of  11 variables:
 $ work_year         : int  2022 2022 2022 2022 2022 2022 2022 2022 2021 2022 ...
 $ experience_level  : chr  "EN" "MI" "MI" "MI" ...
 $ employment_type   : chr  "FT" "FT" "FT" "FT" ...
 $ job_title         : chr  "Cyber Program Manager" "Security Analyst" "Security Analyst" "IT Security Analyst" ...
 $ salary            : int  63000 95000 70000 250000 120000 315000 220000 140000 55000 360000 ...
 $ salary_currency   : chr  "USD" "USD" "USD" "BRL" ...
 $ salary_in_usd     : int  63000 95000 70000 48853 120000 315000 220000 140000 75650 360000 ...
 $ employee_residence: chr  "US" "US" "US" "BR" ...
 $ remote_ratio      : int  50 0 0 50 100 100 100 100 50 100 ...
 $ company_location  : chr  "US" "US" "US" "BR" ...
 $ company_size      : chr  "S" "M" "M" "L" ...

Attributes’ description table

Attribute Name Description Data Type Possible values
work_year The year in which salary was recorded Numerical 2020 to 2022
experience_level Expertise level of the employee Ordinal En “Entry level”
MI “Mid level”
SE “Senior level”
EX “Executive level”
employment_type The nature or category of employee’s engagement in the job Nominal PT “Part time”
FT “Full time”
CT “Contract
FL”Freelancer”
job_title The role worked in during the year Nominal

Different titles.

like Security Analyst, security researcher

salary The total gross salary amount paid Numerical 1740-50001566
salary_currency The currency of the salary paid to the employee Nominal

Different currencies according to ISO 4217 currency code.

like DE,CA

salary_in_usd The salary paid in United states dollar Numerical 2000 to 365596.40
employee_residence Employee’s primary country of residence Nominal

Different countries.

like US,AE

remote_ratio Percentage of online work by employee in the specified year Numerical 0 “No remote work”
50 “Partially remote”
100 “Fully remote”
company_location The country of the employer’s main office Nominal

Different countries.

like BR,BW

company_size How big/small is the company Ordinal S , M or L

phase 2

sample of 20 employees from the dataset:

using sample_n(table,size) function and using (set_seed())

set.seed(30)
sample=sample_n(dataset,20)
print(sample)

Show the missing value:

if it is FALSE it means no null value,if it is TRUE there is null value. In our dataset there is no null values.

is.na(dataset)
        work_year experience_level employment_type job_title salary
   [1,]     FALSE            FALSE           FALSE     FALSE  FALSE
   [2,]     FALSE            FALSE           FALSE     FALSE  FALSE
   [3,]     FALSE            FALSE           FALSE     FALSE  FALSE
   [4,]     FALSE            FALSE           FALSE     FALSE  FALSE
   [5,]     FALSE            FALSE           FALSE     FALSE  FALSE
   [6,]     FALSE            FALSE           FALSE     FALSE  FALSE
   [7,]     FALSE            FALSE           FALSE     FALSE  FALSE
   [8,]     FALSE            FALSE           FALSE     FALSE  FALSE
   [9,]     FALSE            FALSE           FALSE     FALSE  FALSE
  [10,]     FALSE            FALSE           FALSE     FALSE  FALSE
  [11,]     FALSE            FALSE           FALSE     FALSE  FALSE
  [12,]     FALSE            FALSE           FALSE     FALSE  FALSE
  [13,]     FALSE            FALSE           FALSE     FALSE  FALSE
  [14,]     FALSE            FALSE           FALSE     FALSE  FALSE
  [15,]     FALSE            FALSE           FALSE     FALSE  FALSE
  [16,]     FALSE            FALSE           FALSE     FALSE  FALSE
  [17,]     FALSE            FALSE           FALSE     FALSE  FALSE
  [18,]     FALSE            FALSE           FALSE     FALSE  FALSE
  [19,]     FALSE            FALSE           FALSE     FALSE  FALSE
  [20,]     FALSE            FALSE           FALSE     FALSE  FALSE
  [21,]     FALSE            FALSE           FALSE     FALSE  FALSE
  [22,]     FALSE            FALSE           FALSE     FALSE  FALSE
  [23,]     FALSE            FALSE           FALSE     FALSE  FALSE
  [24,]     FALSE            FALSE           FALSE     FALSE  FALSE
  [25,]     FALSE            FALSE           FALSE     FALSE  FALSE
  [26,]     FALSE            FALSE           FALSE     FALSE  FALSE
  [27,]     FALSE            FALSE           FALSE     FALSE  FALSE
  [28,]     FALSE            FALSE           FALSE     FALSE  FALSE
  [29,]     FALSE            FALSE           FALSE     FALSE  FALSE
  [30,]     FALSE            FALSE           FALSE     FALSE  FALSE
  [31,]     FALSE            FALSE           FALSE     FALSE  FALSE
  [32,]     FALSE            FALSE           FALSE     FALSE  FALSE
  [33,]     FALSE            FALSE           FALSE     FALSE  FALSE
  [34,]     FALSE            FALSE           FALSE     FALSE  FALSE
  [35,]     FALSE            FALSE           FALSE     FALSE  FALSE
  [36,]     FALSE            FALSE           FALSE     FALSE  FALSE
  [37,]     FALSE            FALSE           FALSE     FALSE  FALSE
  [38,]     FALSE            FALSE           FALSE     FALSE  FALSE
  [39,]     FALSE            FALSE           FALSE     FALSE  FALSE
  [40,]     FALSE            FALSE           FALSE     FALSE  FALSE
  [41,]     FALSE            FALSE           FALSE     FALSE  FALSE
  [42,]     FALSE            FALSE           FALSE     FALSE  FALSE
  [43,]     FALSE            FALSE           FALSE     FALSE  FALSE
  [44,]     FALSE            FALSE           FALSE     FALSE  FALSE
  [45,]     FALSE            FALSE           FALSE     FALSE  FALSE
  [46,]     FALSE            FALSE           FALSE     FALSE  FALSE
  [47,]     FALSE            FALSE           FALSE     FALSE  FALSE
  [48,]     FALSE            FALSE           FALSE     FALSE  FALSE
  [49,]     FALSE            FALSE           FALSE     FALSE  FALSE
  [50,]     FALSE            FALSE           FALSE     FALSE  FALSE
  [51,]     FALSE            FALSE           FALSE     FALSE  FALSE
  [52,]     FALSE            FALSE           FALSE     FALSE  FALSE
  [53,]     FALSE            FALSE           FALSE     FALSE  FALSE
  [54,]     FALSE            FALSE           FALSE     FALSE  FALSE
  [55,]     FALSE            FALSE           FALSE     FALSE  FALSE
  [56,]     FALSE            FALSE           FALSE     FALSE  FALSE
  [57,]     FALSE            FALSE           FALSE     FALSE  FALSE
  [58,]     FALSE            FALSE           FALSE     FALSE  FALSE
  [59,]     FALSE            FALSE           FALSE     FALSE  FALSE
  [60,]     FALSE            FALSE           FALSE     FALSE  FALSE
  [61,]     FALSE            FALSE           FALSE     FALSE  FALSE
  [62,]     FALSE            FALSE           FALSE     FALSE  FALSE
  [63,]     FALSE            FALSE           FALSE     FALSE  FALSE
  [64,]     FALSE            FALSE           FALSE     FALSE  FALSE
  [65,]     FALSE            FALSE           FALSE     FALSE  FALSE
  [66,]     FALSE            FALSE           FALSE     FALSE  FALSE
  [67,]     FALSE            FALSE           FALSE     FALSE  FALSE
  [68,]     FALSE            FALSE           FALSE     FALSE  FALSE
  [69,]     FALSE            FALSE           FALSE     FALSE  FALSE
  [70,]     FALSE            FALSE           FALSE     FALSE  FALSE
  [71,]     FALSE            FALSE           FALSE     FALSE  FALSE
  [72,]     FALSE            FALSE           FALSE     FALSE  FALSE
  [73,]     FALSE            FALSE           FALSE     FALSE  FALSE
  [74,]     FALSE            FALSE           FALSE     FALSE  FALSE
  [75,]     FALSE            FALSE           FALSE     FALSE  FALSE
  [76,]     FALSE            FALSE           FALSE     FALSE  FALSE
  [77,]     FALSE            FALSE           FALSE     FALSE  FALSE
  [78,]     FALSE            FALSE           FALSE     FALSE  FALSE
  [79,]     FALSE            FALSE           FALSE     FALSE  FALSE
  [80,]     FALSE            FALSE           FALSE     FALSE  FALSE
  [81,]     FALSE            FALSE           FALSE     FALSE  FALSE
  [82,]     FALSE            FALSE           FALSE     FALSE  FALSE
  [83,]     FALSE            FALSE           FALSE     FALSE  FALSE
  [84,]     FALSE            FALSE           FALSE     FALSE  FALSE
  [85,]     FALSE            FALSE           FALSE     FALSE  FALSE
  [86,]     FALSE            FALSE           FALSE     FALSE  FALSE
  [87,]     FALSE            FALSE           FALSE     FALSE  FALSE
  [88,]     FALSE            FALSE           FALSE     FALSE  FALSE
  [89,]     FALSE            FALSE           FALSE     FALSE  FALSE
  [90,]     FALSE            FALSE           FALSE     FALSE  FALSE
        salary_currency salary_in_usd employee_residence remote_ratio
   [1,]           FALSE         FALSE              FALSE        FALSE
   [2,]           FALSE         FALSE              FALSE        FALSE
   [3,]           FALSE         FALSE              FALSE        FALSE
   [4,]           FALSE         FALSE              FALSE        FALSE
   [5,]           FALSE         FALSE              FALSE        FALSE
   [6,]           FALSE         FALSE              FALSE        FALSE
   [7,]           FALSE         FALSE              FALSE        FALSE
   [8,]           FALSE         FALSE              FALSE        FALSE
   [9,]           FALSE         FALSE              FALSE        FALSE
  [10,]           FALSE         FALSE              FALSE        FALSE
  [11,]           FALSE         FALSE              FALSE        FALSE
  [12,]           FALSE         FALSE              FALSE        FALSE
  [13,]           FALSE         FALSE              FALSE        FALSE
  [14,]           FALSE         FALSE              FALSE        FALSE
  [15,]           FALSE         FALSE              FALSE        FALSE
  [16,]           FALSE         FALSE              FALSE        FALSE
  [17,]           FALSE         FALSE              FALSE        FALSE
  [18,]           FALSE         FALSE              FALSE        FALSE
  [19,]           FALSE         FALSE              FALSE        FALSE
  [20,]           FALSE         FALSE              FALSE        FALSE
  [21,]           FALSE         FALSE              FALSE        FALSE
  [22,]           FALSE         FALSE              FALSE        FALSE
  [23,]           FALSE         FALSE              FALSE        FALSE
  [24,]           FALSE         FALSE              FALSE        FALSE
  [25,]           FALSE         FALSE              FALSE        FALSE
  [26,]           FALSE         FALSE              FALSE        FALSE
  [27,]           FALSE         FALSE              FALSE        FALSE
  [28,]           FALSE         FALSE              FALSE        FALSE
  [29,]           FALSE         FALSE              FALSE        FALSE
  [30,]           FALSE         FALSE              FALSE        FALSE
  [31,]           FALSE         FALSE              FALSE        FALSE
  [32,]           FALSE         FALSE              FALSE        FALSE
  [33,]           FALSE         FALSE              FALSE        FALSE
  [34,]           FALSE         FALSE              FALSE        FALSE
  [35,]           FALSE         FALSE              FALSE        FALSE
  [36,]           FALSE         FALSE              FALSE        FALSE
  [37,]           FALSE         FALSE              FALSE        FALSE
  [38,]           FALSE         FALSE              FALSE        FALSE
  [39,]           FALSE         FALSE              FALSE        FALSE
  [40,]           FALSE         FALSE              FALSE        FALSE
  [41,]           FALSE         FALSE              FALSE        FALSE
  [42,]           FALSE         FALSE              FALSE        FALSE
  [43,]           FALSE         FALSE              FALSE        FALSE
  [44,]           FALSE         FALSE              FALSE        FALSE
  [45,]           FALSE         FALSE              FALSE        FALSE
  [46,]           FALSE         FALSE              FALSE        FALSE
  [47,]           FALSE         FALSE              FALSE        FALSE
  [48,]           FALSE         FALSE              FALSE        FALSE
  [49,]           FALSE         FALSE              FALSE        FALSE
  [50,]           FALSE         FALSE              FALSE        FALSE
  [51,]           FALSE         FALSE              FALSE        FALSE
  [52,]           FALSE         FALSE              FALSE        FALSE
  [53,]           FALSE         FALSE              FALSE        FALSE
  [54,]           FALSE         FALSE              FALSE        FALSE
  [55,]           FALSE         FALSE              FALSE        FALSE
  [56,]           FALSE         FALSE              FALSE        FALSE
  [57,]           FALSE         FALSE              FALSE        FALSE
  [58,]           FALSE         FALSE              FALSE        FALSE
  [59,]           FALSE         FALSE              FALSE        FALSE
  [60,]           FALSE         FALSE              FALSE        FALSE
  [61,]           FALSE         FALSE              FALSE        FALSE
  [62,]           FALSE         FALSE              FALSE        FALSE
  [63,]           FALSE         FALSE              FALSE        FALSE
  [64,]           FALSE         FALSE              FALSE        FALSE
  [65,]           FALSE         FALSE              FALSE        FALSE
  [66,]           FALSE         FALSE              FALSE        FALSE
  [67,]           FALSE         FALSE              FALSE        FALSE
  [68,]           FALSE         FALSE              FALSE        FALSE
  [69,]           FALSE         FALSE              FALSE        FALSE
  [70,]           FALSE         FALSE              FALSE        FALSE
  [71,]           FALSE         FALSE              FALSE        FALSE
  [72,]           FALSE         FALSE              FALSE        FALSE
  [73,]           FALSE         FALSE              FALSE        FALSE
  [74,]           FALSE         FALSE              FALSE        FALSE
  [75,]           FALSE         FALSE              FALSE        FALSE
  [76,]           FALSE         FALSE              FALSE        FALSE
  [77,]           FALSE         FALSE              FALSE        FALSE
  [78,]           FALSE         FALSE              FALSE        FALSE
  [79,]           FALSE         FALSE              FALSE        FALSE
  [80,]           FALSE         FALSE              FALSE        FALSE
  [81,]           FALSE         FALSE              FALSE        FALSE
  [82,]           FALSE         FALSE              FALSE        FALSE
  [83,]           FALSE         FALSE              FALSE        FALSE
  [84,]           FALSE         FALSE              FALSE        FALSE
  [85,]           FALSE         FALSE              FALSE        FALSE
  [86,]           FALSE         FALSE              FALSE        FALSE
  [87,]           FALSE         FALSE              FALSE        FALSE
  [88,]           FALSE         FALSE              FALSE        FALSE
  [89,]           FALSE         FALSE              FALSE        FALSE
  [90,]           FALSE         FALSE              FALSE        FALSE
        company_location company_size
   [1,]            FALSE        FALSE
   [2,]            FALSE        FALSE
   [3,]            FALSE        FALSE
   [4,]            FALSE        FALSE
   [5,]            FALSE        FALSE
   [6,]            FALSE        FALSE
   [7,]            FALSE        FALSE
   [8,]            FALSE        FALSE
   [9,]            FALSE        FALSE
  [10,]            FALSE        FALSE
  [11,]            FALSE        FALSE
  [12,]            FALSE        FALSE
  [13,]            FALSE        FALSE
  [14,]            FALSE        FALSE
  [15,]            FALSE        FALSE
  [16,]            FALSE        FALSE
  [17,]            FALSE        FALSE
  [18,]            FALSE        FALSE
  [19,]            FALSE        FALSE
  [20,]            FALSE        FALSE
  [21,]            FALSE        FALSE
  [22,]            FALSE        FALSE
  [23,]            FALSE        FALSE
  [24,]            FALSE        FALSE
  [25,]            FALSE        FALSE
  [26,]            FALSE        FALSE
  [27,]            FALSE        FALSE
  [28,]            FALSE        FALSE
  [29,]            FALSE        FALSE
  [30,]            FALSE        FALSE
  [31,]            FALSE        FALSE
  [32,]            FALSE        FALSE
  [33,]            FALSE        FALSE
  [34,]            FALSE        FALSE
  [35,]            FALSE        FALSE
  [36,]            FALSE        FALSE
  [37,]            FALSE        FALSE
  [38,]            FALSE        FALSE
  [39,]            FALSE        FALSE
  [40,]            FALSE        FALSE
  [41,]            FALSE        FALSE
  [42,]            FALSE        FALSE
  [43,]            FALSE        FALSE
  [44,]            FALSE        FALSE
  [45,]            FALSE        FALSE
  [46,]            FALSE        FALSE
  [47,]            FALSE        FALSE
  [48,]            FALSE        FALSE
  [49,]            FALSE        FALSE
  [50,]            FALSE        FALSE
  [51,]            FALSE        FALSE
  [52,]            FALSE        FALSE
  [53,]            FALSE        FALSE
  [54,]            FALSE        FALSE
  [55,]            FALSE        FALSE
  [56,]            FALSE        FALSE
  [57,]            FALSE        FALSE
  [58,]            FALSE        FALSE
  [59,]            FALSE        FALSE
  [60,]            FALSE        FALSE
  [61,]            FALSE        FALSE
  [62,]            FALSE        FALSE
  [63,]            FALSE        FALSE
  [64,]            FALSE        FALSE
  [65,]            FALSE        FALSE
  [66,]            FALSE        FALSE
  [67,]            FALSE        FALSE
  [68,]            FALSE        FALSE
  [69,]            FALSE        FALSE
  [70,]            FALSE        FALSE
  [71,]            FALSE        FALSE
  [72,]            FALSE        FALSE
  [73,]            FALSE        FALSE
  [74,]            FALSE        FALSE
  [75,]            FALSE        FALSE
  [76,]            FALSE        FALSE
  [77,]            FALSE        FALSE
  [78,]            FALSE        FALSE
  [79,]            FALSE        FALSE
  [80,]            FALSE        FALSE
  [81,]            FALSE        FALSE
  [82,]            FALSE        FALSE
  [83,]            FALSE        FALSE
  [84,]            FALSE        FALSE
  [85,]            FALSE        FALSE
  [86,]            FALSE        FALSE
  [87,]            FALSE        FALSE
  [88,]            FALSE        FALSE
  [89,]            FALSE        FALSE
  [90,]            FALSE        FALSE
 [ reached getOption("max.print") -- omitted 1157 rows ]
sum(is.na(dataset))
[1] 0

Show the Min.,1st Qu.,Median,Mean ,3rd Qu.,Max. for each numeric column

The summary statistics for the dataset variables provide insights into the distribution of features. we can conclude the following:

In work_year: The data spans from the year 2020 to 2022 with Most data falling within the years 2021 and 2022, as indicated by both the median and mean being centered around 2021.

In salary: Salaries vary widely with a minimum of $1,740 and a maximum of $500 million. The median is $120,000 which is a mid value, but the mean is notably higher at $560,852 which might be duo to extreme values or notable skewness.

In salary_in_usd: The data has a median of $110,000, and a mean of $120,278, and the spread of salaries is observable in the difference between the median and mean.

In remote_ratio: Indicates the percentage of remote work ranging from 0% to 100%, with a median and 3rd quartile at 100%, and a mean of 71.49%, indicating a notable presence of remote work in the dataset, suggesting some variability.

summary(dataset$work_year)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
   2020    2021    2021    2021    2022    2022 
summary(dataset$salary)
     Min.   1st Qu.    Median      Mean   3rd Qu.      Max. 
     1740     79754    120000    560852    160080 500000000 
summary(dataset$salary_in_usd)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
   2000   74595  110000  120278  150000  910991 
summary(dataset$remote_ratio)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
   0.00   50.00  100.00   71.49  100.00  100.00 

Show the variane of each numeric column

variance is to understand the spread or dispersion of the values in each column. A higher variance indicates that the values are more spread out from the mean and in our dataset the highest varied attribute is salary, while a lower variance indicates that the values are closer to the mean which in our datas it is work year attribute.

Variance results reveal that: -work years are to some extent consistent -salaries show notable variability and possible outliers -salaries in USD have a stable distribution -remote work ratio have moderate variability

var(dataset$work_year)
[1] 0.511942
var(dataset$salary)
[1] 2.004897e+14
var(dataset$salary_in_usd)
[1] 4940880203
var(dataset$remote_ratio)
[1] 1548.175

Visualization of relationship between some pairs of attributes:

Here we used boxplot to see the distribution between salary_in_usd and experience_level We observed that salaries vary depending on the level of experience,they are positively correlated.

boxplot(salary_in_usd ~ experience_level, data = dataset , yaxt="n")
labels<- pretty(dataset$salary_in_usd)
labels<- sapply(labels, function(x) format(x, scientific = FALSE))
axis(side = 2, at=pretty(dataset$salary_in_usd), labels = labels )

options(scipen = 999)

Here we used boxplot to see the distribution between salary_in_usd and work_year We observed that 2021 salaries were close to each other but in 2022 the gap between them getting bigger.

boxplot(salary_in_usd ~ work_year, data = dataset , yaxt="n")
labels<- pretty(dataset$salary_in_usd)
labels<- sapply(labels, function(x) format(x, scientific = FALSE))
axis(side = 2, at=pretty(dataset$salary_in_usd), labels = labels )

options(scipen = 999)

Here we used boxplot to see the distribution between salary_in_usd and employment_type We observed that Full Time (FT) offers more salary than the other categories.

boxplot(salary_in_usd ~ employment_type, data = dataset , yaxt="n")
labels<- pretty(dataset$salary_in_usd)
labels<- sapply(labels, function(x) format(x, scientific = FALSE))
axis(side = 2, at=pretty(dataset$salary_in_usd), labels = labels )

options(scipen = 999)

Here we used boxplot to see the distribution between salary_in_usd and company_size We observed that the larger the company is the higher the salary was.

boxplot(salary_in_usd ~ company_size, data = dataset , yaxt="n")
labels<- pretty(dataset$salary_in_usd)
labels<- sapply(labels, function(x) format(x, scientific = FALSE))
axis(side = 2, at=pretty(dataset$salary_in_usd), labels = labels )

options(scipen = 999) 

Data Reduction

Dimensionality Reduction

The “salary” column gives the same information as “salary_in_usd” it’s just a matter of currency exchange, and we will eventually transform all the values in “salary” column to one common currency so we can properly deal with them. To further confirm that the two column are redundant, we will use the latest exchange rate for USD to the desired currency.

we will start by creating a temporary column named “converted_salary” to save the salary that we will get by using the exchange rate to convert the salary_in_usd to the salary with different currencies to compare with “salary” column

convertedDataset=dataset


convertedDataset$exchange_rate = factor(convertedDataset$salary_currency, levels=c("USD","BRL","GBP","EUR","INR","CAD","CHF","DKK","SGD","AUD","SEK","MXN","ILS","PLN","NOK","IDR","NZD","HUF","ZAR","TWD","RUB"), labels=c(1/1,1/0.20,1/1.22,1/1.06,1/0.012,1/0.74,1/1.10,1/0.14,1/0.73,1/0.64,1/0.090,1/0.057,1/0.26,1/0.23,1/0.093,1/0.000065,1/0.60,1/0.0027,1/0.053,1/0.031,1/0.010))
convertedDataset$exchange_rate = as.numeric(as.character(convertedDataset$exchange_rate))
convertedDataset$converted_salary = convertedDataset$salary_in_usd*convertedDataset$exchange_rate



set.seed(1)
salary_sample <- sample_n(convertedDataset[,c("salary","converted_salary")],10)

print(salary_sample)

as shown in the sample, the two columns are almost identical. This can be proved by correlation coefficient as well.

correlation <- cor(convertedDataset$salary , convertedDataset$converted_salary)
print(correlation)
[1] 0.9999943

The correlation is so high but it hasn’t reached 100% possibly due to rounding in the calculations and slight differences in the exchange rate over time.

To make the mining process more effiecent and has an improved quality, we decided to remove the “salary” column.

dataset<-dataset[,-c(5)]

Find the outliers and remove them:

We will show outliers with boxPlots and then remove them, to minimize noise and to get better analytical results when applying data mining techniques.

now we show (salary_in_usd) attributes’ outliers. we can see that there are many outliers with exceptionally high values, thus we will remove them.

boxplot(dataset$salary_in_usd)




OutSalary = outlier(dataset$salary_in_usd, logical =TRUE)
Find_outlier = which(OutSalary ==TRUE, arr.ind = TRUE)
dataset= dataset[-Find_outlier,]

now we show (remote_ratio) attributes’ outliers. we can see there aren’t outliers in remote_ratio, thus we don’t need the last step i.e: removing outliers’ rows.

boxplot(dataset$remote_ratio)

now we show (work_year) attributes’ outliers. we can see there aren’t outliers in work_year, thus we don’t need the last step i.e: removing outliers’ rows.

boxplot(dataset$work_year)

Concept hierarchy generation for nominal data

the columns “company_location” and “employee_residence” have the name of countries for the company and employee respectively. And these attributes can be generalized to higher-level concept that is region to help understand and analyze the dataset better and improve algorithm performance.

We will use the 7 regions as defined in the World Bank Development Indicators. These regions are:

  1. East Asia and Pacific: This region includes countries like China, Australia, Indonesia, Thailand, etc.

  2. Europe and Central Asia: This region includes countries like Germany, UK, Russia, Turkey, etc.

  3. Latin America & Caribbean: This region includes countries like Brazil, Mexico, Argentina, Cuba, etc.

  4. Middle East and North Africa: This region includes countries like Saudi Arabia, Egypt, Iran, Iraq, etc.

  5. North America: This is predominantly United States and Canada.

  6. South Asia: This region includes countries like India, Pakistan, Bangladesh, Sri Lanka, etc.

  7. Sub-Saharan Africa: This region includes countries like Nigeria, South Africa, Ethiopia, Kenya, etc.

Note: UM(The United States Minor Outlying Islands) and AQ(Antarctica) don’t belong to any of these regions, thus, they will be used as they are.



um=which(dataset$company_location=="UM")
aq=which(dataset$company_location=="AQ")


dataset$company_location <- countrycode(dataset$company_location, "iso2c", "region")
Warning: Some values were not matched unambiguously: AQ, UM
dataset$employee_residence <- countrycode(dataset$employee_residence, "iso2c", "region")

dataset[um,"company_location"]="UM"
dataset[aq,"company_location"]="AQ"

Concept hierarchy generation can be done for “job_title” as well to improve interpretation and scalability. Also, most job titles are essentially the same job but with different names, so we can combine them into a higher-level jobs titles such as Architect, Analyst and Engineer etc.

## Create the categories based on job rank 
dataset$job_title <- ifelse(grepl("Analyst", dataset$job_title), "Analyst",
                                ifelse(grepl("Architect", dataset$job_title), "Architect",
                                       ifelse(grepl("Engineer", dataset$job_title), "Engineer",
                                              ifelse(grepl("Manager|Officer|Director|Leader", dataset$job_title), "Leadership",
                                                     ifelse(grepl("Consultant|Specialist", dataset$job_title), "Consultant/Specialist",
                                                            ifelse(grepl("Cyber", dataset$job_title), "Cyber Security",
                                                                   "Others"))))))

Encoding categorical data

To deal with columns with character type we are going to encode them, because most machine learning algorithms are designed to work with factors data rather than character data and it improves performance and Interpretability of data as well.

dataset$job_title  <- factor(dataset$job_title)

dataset$experience_level = factor(dataset$experience_level, levels=c("EN", "MI", "SE", "EX"), labels=c(1,2,3,4))

dataset$employment_type  <- factor(dataset$employment_type)

dataset$employee_residence  <- factor(dataset$employee_residence)

dataset$company_location  <- factor(dataset$company_location)

dataset$salary_currency  <- factor(dataset$salary_currency)

dataset$job_title  <- factor(dataset$job_title)


dataset$company_size = factor(dataset$company_size, levels=c("S","M","L"), labels=c(1,2,3))


dataset$job_title  <- factor(dataset$job_title)

Discretization of salaray_in_usd attribute

by calculating breaks based on quartiles

breaks <- quantile(dataset$salary_in_usd, 
                   probs = c(0, .25, .5, .75, .95, 1), 
                   na.rm = TRUE)


dataset$salary_in_usd <- cut(dataset$salary_in_usd, 
                                       breaks = breaks, 
                                       include.lowest = TRUE, 
                                       labels=c("Very Low", "Low", "Medium", "High", "Very High"))

Normalization:

to change the scale of numeric attributes (remote_ratio and work_year) to a scale of [-1,1] to give them equal weight

dataset [, c("work_year" , "remote_ratio")] = scale(dataset [, c("work_year" , "remote_ratio")])

Feature Selection

we will implement feature selection to remove redundant or irrelevant attributes from the data set to get the smallest subset that can help us get the most accurate predictions for our target class(salary_in_usd) and decrease the time that it takes the classifier to process the data.

we will use RFE(Recursive feature elimination) which is a wrapper method for the feature selection. Since the RFE function have multiple control options we need to specify the options that we want. We will choose “Random Forest” because it has high accuracy, can handle categorical data.

control <- rfeControl(functions = rfFuncs, 
                      method = "repeatedcv",
                      repeats = 5, 
                      number = 10)

First we save the features to be used in the feature selection(every attributes except the class label “salary_in_usd”) in variable x, and the class label in variable y. Then split the data to 80% training and 20% test.

x <- dataset %>%
  select(-salary_in_usd) %>%
  as.data.frame()

# Target variable
y <- dataset$salary_in_usd

# Training: 80%; Test: 20%
set.seed(2021)
inTrain <- createDataPartition(y, p = .80, list = FALSE)[,1]

x_train <- x[ inTrain, ]
x_test  <- x[-inTrain, ]

y_train <- y[ inTrain]
y_test  <- y[-inTrain]

after splitting the data, now we can perform the selection using rfe

set.seed(1)
result_rfe1 <- rfe(x = x_train, 
                   y = y_train, 
                   sizes = c(1:9),
                   rfeControl = control)

result_rfe1

Recursive feature selection

Outer resampling method: Cross-Validated (10 fold, repeated 5 times) 

Resampling performance over subset size:

The top 5 variables (out of 8):
   experience_level, employee_residence, salary_currency, company_location, work_year
predictors(result_rfe1)
[1] "experience_level"   "employee_residence" "salary_currency"   
[4] "company_location"   "work_year"          "job_title"         
[7] "company_size"       "remote_ratio"      

The results show that all the remaining attributes, except for “employment_type”, are selected. This is logical, as 98% of the rows have the value “FT”, as shown in the table below. Due to the low variance, we decided to remove this attribute.

table(dataset$employment_type)

  CT   FL   FT   PT 
  11    3 1224    8 
dataset<-dataset[,-which(names(dataset)=="employment_type")]

phase 3

During this phase, our focus will be on clustering and classification techniques to analyze the data. The primary objectives are to identify distinct groups within the dataset through clustering, classify data objects into meaningful categories, and apply different evaluation methods to assess the accuracy and precision of both classification and clustering results. We aim to gain deeper insights into the data and discover patterns.

Retreive our preprocessed dataset


# Read the CSV file from github
dataset2= read.csv(url("https://raw.githubusercontent.com/SarahAlhindi/DM_project/main/Data%20Set/preprocessedDataset.csv"), header=TRUE)

# Identify the character variables in the dataset2
char_vars <- sapply(dataset2, is.character)

# Convert the identified character variables in dataset2 to factors
dataset2[char_vars] <- lapply(dataset2[char_vars], as.factor)

balancing data

To resolve the problem of class imbalance in the dataset, we will use SMOTE() method that oversample the minority class by creating synthetic samples using the existing minority class samples

set.seed(10)
balanced_dataset <- SMOTE(salary_in_usd ~ ., dataset2, perc.over = 300, perc.under=500, k = 10)

Classification

The goal of all preceding steps is to properly prepare the dataset for the classification and later clustering phase, which constitutes one of our primary mining objectives. In this section, we will employ various attribute selection methods such as the Gini index, Gain ratio, and information gain to construct a decision tree model. We will thoroughly evaluate its performance, and if it proves effective, it can subsequently be utilized to classify new instances with unknown class labels.

since our dataset is small, we decided to use K-fold Cross-validation as a dataset partioning method. for each attribute selection method we will try different K size (10,5, and 3)

in all this section we will be using train and trainControl functions of caret package to produce decision trees. for Gini index the method will be “rpart” and for Gain ratio it’s “j48” as for information gain the method is “C5.0”.

the following function will be used to compute average sensitivity and Specificity:



macro = function(matrix){
  
  sumSen=0
  
  for (i in 1:5) {
   sumSen = sumSen + matrix$byClass[i,1] 
  }
  
  
  avgSen = sumSen/5
  
  sumSpec=0
  
  for (i in 1:5) {
   sumSpec = sumSpec + matrix$byClass[i,2] 
  }
  avgSpec = sumSpec/5
  
  
  
  
  sumPrec=0
  
  for (i in 1:5) {
   sumPrec = sumPrec + matrix$byClass[i,3] 
  }
  avgPrec = sumPrec/5
  
  
  
  
  avgs = data.frame(Sensitivity=avgSen , Specificity=avgSpec, Precision=avgPrec ,Accuracy= unname( matrix$overall[1]) )
  print(avgs)
  
  
}

Gini index

Gini index measures the impurity of the dataset. The partitioning that yields the most substantial reduction in impurity is selected as the splitting attribute. To apply the Gini index, we will employ the “rpart” method, which utilizes the Gini index as the criteria for splitting.

10 Folds

The tree of the gini index using 10 folds


set.seed(10)
ctrl <- trainControl(method = "cv", number = 10, returnResamp="all", savePredictions="final")

tuneGrid <- expand.grid(cp = c(0.001, 0.005, 0.01))

giniIndex10 <- train(
  salary_in_usd ~ .,
  data = balanced_dataset,
  method = "rpart",
  trControl = ctrl,tuneGrid=tuneGrid,
  control = list(
    minsplit = 10,
    minbucket = 5,
    xval = 10,
    cp = 0.0001
  )

)


prp(giniIndex10$finalModel, box.palette = "Reds", tweak = 1.2, varlen = 20)

the “experince level” attribute was selected as the first splitting attribute meaning that it has the largest impurity reduction.

Confusion matrix of 10 folds using Gini Index

The following confusion Matrix will show the performance of the classifier using the predicted class labels and the actual class labels of our dataset


giniIndex10cm = caret::confusionMatrix(giniIndex10$pred$obs,giniIndex10$pred$pred)

giniIndex10cm
Confusion Matrix and Statistics

           Reference
Prediction  High Low Medium Very_High Very_Low
  High        99  16     37        19        3
  Low         28 130     42         1       69
  Medium      56  64     88        15       14
  Very_High   28   5     19       190       10
  Very_Low     6  54      6         9      189

Overall Statistics
                                              
               Accuracy : 0.5815              
                 95% CI : (0.5529, 0.6096)    
    No Information Rate : 0.2381              
    P-Value [Acc > NIR] : < 0.0000000000000002
                                              
                  Kappa : 0.4752              
                                              
 Mcnemar's Test P-Value : 0.01206             

Statistics by Class:

                     Class: High Class: Low Class: Medium
Sensitivity              0.45622     0.4833       0.45833
Specificity              0.92347     0.8491       0.85174
Pos Pred Value           0.56897     0.4815       0.37131
Neg Pred Value           0.88465     0.8501       0.89167
Prevalence               0.18129     0.2247       0.16040
Detection Rate           0.08271     0.1086       0.07352
Detection Prevalence     0.14536     0.2256       0.19799
Balanced Accuracy        0.68985     0.6662       0.65504
                     Class: Very_High Class: Very_Low
Sensitivity                    0.8120          0.6632
Specificity                    0.9356          0.9178
Pos Pred Value                 0.7540          0.7159
Neg Pred Value                 0.9534          0.8971
Prevalence                     0.1955          0.2381
Detection Rate                 0.1587          0.1579
Detection Prevalence           0.2105          0.2206
Balanced Accuracy              0.8738          0.7905

the metrics shown for each class indicate the value of that metric when treating this class as the positive class and the other classes as the negative class. here the classifier showed best performance when using the “very high” class as the positive class but this value in its own doesn’t hold much value since all classes should be taken into consideration.

5 Folds

The tree of the gini index using 5 folds

set.seed(10)
ctrl <- trainControl(method = "cv", number = 5, returnResamp="all", savePredictions="final")


tuneGrid <- expand.grid(cp = c(0.001, 0.005, 0.01))

giniIndex5 <- train(salary_in_usd ~ ., data = balanced_dataset, method = "rpart", trControl = ctrl,tuneGrid=tuneGrid,
  control = list(
    minsplit = 10,
    minbucket = 5,
    xval = 10,
    cp = 0.0001
  ))

prp(giniIndex5$finalModel, box.palette = "Reds", tweak = 1.5, varlen = 10, cex = 0.15)

NA
NA

this tree has the same structure as the previous tree that used 10 folds. so in this tree as well “experience level” was choose as the first splitting attribute

Confusion matrix of 5 folds using Gini Index

The following confusion Matrix will show the performance of the classifier using the predicted class labels and the actual class labels of our dataset

giniIndex5cm = caret::confusionMatrix(giniIndex5$pred$obs,giniIndex5$pred$pred)

giniIndex5cm
Confusion Matrix and Statistics

           Reference
Prediction  High Low Medium Very_High Very_Low
  High        98  10     43        16        7
  Low         28 123     48         0       71
  Medium      64  56     87        14       16
  Very_High   25   7     22       192        6
  Very_Low     4  53     10         7      190

Overall Statistics
                                               
               Accuracy : 0.5764               
                 95% CI : (0.5479, 0.6046)     
    No Information Rate : 0.2423               
    P-Value [Acc > NIR] : < 0.00000000000000022
                                               
                  Kappa : 0.4692               
                                               
 Mcnemar's Test P-Value : 0.001289             

Statistics by Class:

                     Class: High Class: Low Class: Medium
Sensitivity              0.44749     0.4940       0.41429
Specificity              0.92229     0.8449       0.84802
Pos Pred Value           0.56322     0.4556       0.36709
Neg Pred Value           0.88172     0.8641       0.87188
Prevalence               0.18296     0.2080       0.17544
Detection Rate           0.08187     0.1028       0.07268
Detection Prevalence     0.14536     0.2256       0.19799
Balanced Accuracy        0.68489     0.6695       0.63116
                     Class: Very_High Class: Very_Low
Sensitivity                    0.8384          0.6552
Specificity                    0.9380          0.9184
Pos Pred Value                 0.7619          0.7197
Neg Pred Value                 0.9608          0.8928
Prevalence                     0.1913          0.2423
Detection Rate                 0.1604          0.1587
Detection Prevalence           0.2105          0.2206
Balanced Accuracy              0.8882          0.7868

the results are very close to the 10 folds tree, so here as well the classifier shows better performance when dealing with the “very high”

3 Folds

The tree of the gini index using 3 folds

set.seed(10)
ctrl <- trainControl(method = "cv", number = 3, returnResamp="all", savePredictions="final")


tuneGrid <- expand.grid(cp = c(0.001, 0.005, 0.01))

giniIndex3 <- train(salary_in_usd ~ ., data = balanced_dataset, method = "rpart", trControl = ctrl,tuneGrid=tuneGrid,
  control = list(
    minsplit = 10,
    minbucket = 5,
    xval = 10,
    cp = 0.0001
  ))

prp(giniIndex3$finalModel, box.palette = "Reds", tweak = 1.5, varlen = 10, cex = 0.15)

NA
NA

The tree shows similar structure as the two previous two trees, whether it’s in its choose of the splitting attributes or the leaves.

Confusion matrix of 3 folds using Gini Index

The following confusion Matrix will show the performance of the classifier using the predicted class labels and the actual class labels of our dataset


giniIndex3cm = caret::confusionMatrix(giniIndex3$pred$obs,giniIndex3$pred$pred)

giniIndex3cm
Confusion Matrix and Statistics

           Reference
Prediction  High Low Medium Very_High Very_Low
  High        99   9     39        19        8
  Low         26 114     48         9       73
  Medium      62  46     86        20       23
  Very_High   31   6     20       189        6
  Very_Low     8  50      9        14      183

Overall Statistics
                                               
               Accuracy : 0.5606               
                 95% CI : (0.5319, 0.5889)     
    No Information Rate : 0.2448               
    P-Value [Acc > NIR] : < 0.00000000000000022
                                               
                  Kappa : 0.4498               
                                               
 Mcnemar's Test P-Value : 0.0006718            

Statistics by Class:

                     Class: High Class: Low Class: Medium
Sensitivity              0.43805    0.50667       0.42574
Specificity              0.92276    0.83951       0.84824
Pos Pred Value           0.56897    0.42222       0.36287
Neg Pred Value           0.87586    0.88026       0.87917
Prevalence               0.18881    0.18797       0.16876
Detection Rate           0.08271    0.09524       0.07185
Detection Prevalence     0.14536    0.22556       0.19799
Balanced Accuracy        0.68041    0.67309       0.63699
                     Class: Very_High Class: Very_Low
Sensitivity                    0.7530          0.6246
Specificity                    0.9334          0.9104
Pos Pred Value                 0.7500          0.6932
Neg Pred Value                 0.9344          0.8821
Prevalence                     0.2097          0.2448
Detection Rate                 0.1579          0.1529
Detection Prevalence           0.2105          0.2206
Balanced Accuracy              0.8432          0.7675

here as well the “very high” class has the best overall performance

Analysis of the gini index classification

All three trees seem to be alike in their arrangement and form.

  1. Root Node - Experience Level: The initial attribute used for splitting the dataset at the root node is the “experience level.” This divides the tree into two main branches or subtrees:
    • Right Subtree: This comprises instances with Senior (SE) and Executive (EX) experience levels.
    • Left Subtree: This includes individuals with Entry (EN) and Mid (MI) experience levels.
  2. Right Subtree - work year: The next attribute used to further classify the right subtree is “work year.” The decision criterion is:
    • If work year is <-1.8: Then it is high.
    • If work year is NOT <-1.8: The next attribute examined is “experience level.”
  3. Left Subtree - Experience Level: On the left side of the tree, the attribute “experience level.” is used to further bifurcate the instances:
    • If experience level is >=2: The next attribute examined is “experience level.”
    • If experience level is NOT >=2: The next attribute also will examined is “experience level.”
Sensitivity, Accuracy, Specifity and precision of all 3,5 and 10 folds using Gini Index
rbind("10 Folds"=macro(giniIndex10cm), "5 Folds"=macro(giniIndex5cm), "3 Folds"=macro(giniIndex3cm)  ) 

The higher values for sensitivity, specificity, precision, and accuracy in the 10-fold case indicate better overall performance according to these metrics. so,Gini Index model performs better with a 10-fold cross-validation compared to 5 and 3 folds.

Gain ratio

The gain ratio, a normalized measure of information gain, is calculated by dividing information gain by the split information. The attribute that yields the highest gain ratio is chosen as the splitting attribute. The C4.5 algorithm employs the gain ratio.

The J48 is the Java-based open-source implementation of the C4.5 algorithm, and it is included in the Weka package. This implementation allows users to conveniently apply the C4.5 decision tree.

10 Folds

The tree of the gain ratio using 10 folds

set.seed(10)
ctrl <- trainControl(method = "cv", number = 10, returnResamp="all", savePredictions="final")
gainRatio10 <- train(salary_in_usd ~ ., data = balanced_dataset, method = "J48",trControl = ctrl)
plot(gainRatio10$finalModel)

the first splitting attribute that was choosen is the “Expeirence level” attribute meaning that it probably has a high information gain and low splitInfo(Entropy of distribution of tuples into partition)

Confusion matrix of 10 folds using Gain ratio

The following confusion Matrix will show the performance of the classifier using the predicted class labels and the actual class labels of our dataset

gainRatio10cm = caret::confusionMatrix(gainRatio10$pred$obs, gainRatio10$pred$pred)

gainRatio10cm
Confusion Matrix and Statistics

           Reference
Prediction  High Low Medium Very_High Very_Low
  High       100  18     32        21        3
  Low         28 148     40         3       51
  Medium      53  49    115        14        6
  Very_High   28   6     18       194        6
  Very_Low     2  39     11         5      207

Overall Statistics
                                             
               Accuracy : 0.6383             
                 95% CI : (0.6103, 0.6655)   
    No Information Rate : 0.2281             
    P-Value [Acc > NIR] : <0.0000000000000002
                                             
                  Kappa : 0.5465             
                                             
 Mcnemar's Test P-Value : 0.167              

Statistics by Class:

                     Class: High Class: Low Class: Medium
Sensitivity              0.47393     0.5692       0.53241
Specificity              0.92495     0.8698       0.87564
Pos Pred Value           0.57471     0.5481       0.48523
Neg Pred Value           0.89150     0.8792       0.89479
Prevalence               0.17627     0.2172       0.18045
Detection Rate           0.08354     0.1236       0.09607
Detection Prevalence     0.14536     0.2256       0.19799
Balanced Accuracy        0.69944     0.7195       0.70402
                     Class: Very_High Class: Very_Low
Sensitivity                    0.8186          0.7582
Specificity                    0.9396          0.9383
Pos Pred Value                 0.7698          0.7841
Neg Pred Value                 0.9545          0.9293
Prevalence                     0.1980          0.2281
Detection Rate                 0.1621          0.1729
Detection Prevalence           0.2105          0.2206
Balanced Accuracy              0.8791          0.8483

here the classifier shows better performance when treating “very high” and “very low” attributes as positive class. since the “very high” class is better in Sensitivity and “very low” is better in Specificity and precision (Pos Pred Value)

5 Folds

The tree of the gain ratio using 5 folds

set.seed(10)
ctrl <- trainControl(method = "cv", number = 5, returnResamp="all", savePredictions="final")
gainRatio5 <- train(salary_in_usd ~ ., data = balanced_dataset, method = "J48",trControl = ctrl)
plot(gainRatio5$finalModel)

the tree is similar to the tree that was resulted from 10 folds. it has choose “Experience level” as the first splitting attribute and and seem to show similar behavior.

Confusion matrix of 5 folds using Gain ratio

The following confusion Matrix will show the performance of the classifier using the predicted class labels and the actual class labels of our dataset


gainRatio5cm=caret::confusionMatrix(gainRatio5$pred$obs, gainRatio5$pred$pred)

gainRatio5cm
Confusion Matrix and Statistics

           Reference
Prediction  High Low Medium Very_High Very_Low
  High       102  21     34        16        1
  Low         31 148     38         1       52
  Medium      56  46    103        15       17
  Very_High   30   7     18       194        3
  Very_Low     3  42     10         9      200

Overall Statistics
                                               
               Accuracy : 0.6241               
                 95% CI : (0.5959, 0.6516)     
    No Information Rate : 0.2281               
    P-Value [Acc > NIR] : < 0.00000000000000022
                                               
                  Kappa : 0.5289               
                                               
 Mcnemar's Test P-Value : 0.007667             

Statistics by Class:

                     Class: High Class: Low Class: Medium
Sensitivity              0.45946     0.5606       0.50739
Specificity              0.92615     0.8692       0.86519
Pos Pred Value           0.58621     0.5481       0.43460
Neg Pred Value           0.88270     0.8749       0.89583
Prevalence               0.18546     0.2206       0.16959
Detection Rate           0.08521     0.1236       0.08605
Detection Prevalence     0.14536     0.2256       0.19799
Balanced Accuracy        0.69281     0.7149       0.68629
                     Class: Very_High Class: Very_Low
Sensitivity                    0.8255          0.7326
Specificity                    0.9397          0.9307
Pos Pred Value                 0.7698          0.7576
Neg Pred Value                 0.9566          0.9218
Prevalence                     0.1963          0.2281
Detection Rate                 0.1621          0.1671
Detection Prevalence           0.2105          0.2206
Balanced Accuracy              0.8826          0.8317

unlike the 10 folds, here the classifier has the best overall performance when considering the “very high” as the positive class.

3 Folds

The tree of the gain ratio using 3 folds

set.seed(10)
ctrl <- trainControl(method = "cv", number = 3, returnResamp="all", savePredictions="final")
gainRatio3 <- train(salary_in_usd ~ ., data = balanced_dataset, method = "J48",trControl = ctrl)
plot(gainRatio3$finalModel)

the tree shows similar behavior as the previous 2 trees that resulted from using 10 and 5 folds.

Confusion matrix of 3 folds using Gain ratio

The following confusion Matrix will show the performance of the classifier using the predicted class labels and the actual class labels of our dataset

gainRatio3cm=caret::confusionMatrix(gainRatio3$pred$obs, gainRatio3$pred$pred)

gainRatio3cm
Confusion Matrix and Statistics

           Reference
Prediction  High Low Medium Very_High Very_Low
  High        94  18     39        19        4
  Low         25 129     39         3       74
  Medium      47  47    110        19       14
  Very_High   14   4     27       200        7
  Very_Low     3  32      9        12      208

Overall Statistics
                                               
               Accuracy : 0.619                
                 95% CI : (0.5909, 0.6467)     
    No Information Rate : 0.2565               
    P-Value [Acc > NIR] : < 0.00000000000000022
                                               
                  Kappa : 0.5216               
                                               
 Mcnemar's Test P-Value : 0.007322             

Statistics by Class:

                     Class: High Class: Low Class: Medium
Sensitivity              0.51366     0.5609        0.4911
Specificity              0.92110     0.8542        0.8695
Pos Pred Value           0.54023     0.4778        0.4641
Neg Pred Value           0.91300     0.8910        0.8812
Prevalence               0.15288     0.1921        0.1871
Detection Rate           0.07853     0.1078        0.0919
Detection Prevalence     0.14536     0.2256        0.1980
Balanced Accuracy        0.71738     0.7075        0.6803
                     Class: Very_High Class: Very_Low
Sensitivity                    0.7905          0.6775
Specificity                    0.9449          0.9371
Pos Pred Value                 0.7937          0.7879
Neg Pred Value                 0.9439          0.8939
Prevalence                     0.2114          0.2565
Detection Rate                 0.1671          0.1738
Detection Prevalence           0.2105          0.2206
Balanced Accuracy              0.8677          0.8073

similar to teh 5 folds, the “very high” class has the best metrics.

Analysis of the gain ratio classification

The observed structure of the three decision trees seems to be the same and it can be summarized as follows:

  1. Root Node - Experience Level: The initial attribute used for splitting the dataset at the root node is the “experience level.” This divides the tree into two main branches or subtrees:
    • Right Subtree: This comprises instances with Senior (SE) and Executive (EX) experience levels.
    • Left Subtree: This includes individuals with Entry (EN) and Mid (MI) experience levels.
  2. Within the right subtree:
    • If the ‘experience level’ is 4 (EX, Executive level) , the tree splits based on the ‘Employee_residence’ attribute. It checks whether the ‘Employee_residence’ is ‘Latin America.’
    • If ‘Employee_residence’ does not equal ‘Latin America,’ the differentiation continues with the ‘remote_ratio’ attribute, further dividing the tree.
  3. Within the left subtree:
    • If the ‘experience level’ is 1 (EN, Entry level), the tree divide based on the ‘Employee_residence’ attribute, specifically checking for ‘Sub-Saharan Africa.’
    • If the ‘experience level’ is 2 (MI, Mid level), it also branches based on ‘Employee_residence,’ but in this case, looking to see if it equals ‘North America.’

The decision tree continues to select the most appropriate attributes for splitting at each node, progressively refining the decision process until it reaches the leaves, where final class labels are assigned to the instances.

Sensitivity, Accuracy, Specifity and precision of all 3,5 and 10 folds using Gain ratio
rbind("10 Folds"=macro(gainRatio10cm), "5 Folds"=macro(gainRatio5cm), "3 Folds"=macro(gainRatio3cm)  ) 

Based on the evaluation metrics of average Sensitivity,Precision ,Specificity, and Accuracy, it is evident that the gain ratio model, built using a 10-fold cross-validation approach, exhibits superior performance compared to the other two models. However, it’s worth noting that the difference in performance between the models is relatively small.

A detailed examination of the results from the 10-fold cross-validation reveals that the model has a notably high specificity compared to other metrics. This high specificity suggests that the model is particularly effective at correctly identifying instances that do not pertain to the target class—essentially, it accurately recognizes when examples are not members of the specified class. For example, if the positive class in question is “High” then the model is able to correctly classify tuples that belong to “Very Low”, “Medium”, and “Very High”.

However, possessing high specificity alone does not guarantee the overall effectiveness of the model, as a well-rounded model also requires balanced performance across other metrics. In this case, its ability to capture and classify instances that do belong to the positive class (as measured by sensitivity) is not as robust. For a model to be considered truly effective, it would need to demonstrate strong performance in all metrics specificity and sensitivity, ensuring it can accurately distinguish both negative and positive instances as well as accuracy precision.

Information gain

Information Gain is a metric used to decide which attribute to choose for splitting the data at each node in the decision tree. For a given dataset, the Information Gain of an attribute is calculated by comparing the entropy before and after the dataset is split based on that attribute. The attribute with the highest Information Gain is chosen as the splitting attribute.

10 Folds

The tree of the information gain using 10 folds

set.seed(10)
ctrl <- trainControl(method = "cv", number = 10, returnResamp="all", savePredictions="final")


infoGain10 <- train(salary_in_usd ~ ., data = balanced_dataset, method = "C5.0",trControl = ctrl)
Warning: 'trials' should be <= 8 for this object. Predictions generated using 8 trialsWarning: 'trials' should be <= 4 for this object. Predictions generated using 4 trialsWarning: 'trials' should be <= 7 for this object. Predictions generated using 7 trialsWarning: 'trials' should be <= 5 for this object. Predictions generated using 5 trialsWarning: 'trials' should be <= 6 for this object. Predictions generated using 6 trialsWarning: 'trials' should be <= 7 for this object. Predictions generated using 7 trialsWarning: 'trials' should be <= 4 for this object. Predictions generated using 4 trialsWarning: 'trials' should be <= 9 for this object. Predictions generated using 9 trialsWarning: 'trials' should be <= 5 for this object. Predictions generated using 5 trialsWarning: 'trials' should be <= 6 for this object. Predictions generated using 6 trialsWarning: 'trials' should be <= 4 for this object. Predictions generated using 4 trialsWarning: 'trials' should be <= 9 for this object. Predictions generated using 9 trialsWarning: 'trials' should be <= 8 for this object. Predictions generated using 8 trialsWarning: 'trials' should be <= 8 for this object. Predictions generated using 8 trialsWarning: 'trials' should be <= 6 for this object. Predictions generated using 6 trialsWarning: 'trials' should be <= 9 for this object. Predictions generated using 9 trialsWarning: 'trials' should be <= 7 for this object. Predictions generated using 7 trialsWarning: 'trials' should be <= 8 for this object. Predictions generated using 8 trialsWarning: 'trials' should be <= 8 for this object. Predictions generated using 8 trialsWarning: 'trials' should be <= 5 for this object. Predictions generated using 5 trialsWarning: 'trials' should be <= 7 for this object. Predictions generated using 7 trialsWarning: 'trials' should be <= 7 for this object. Predictions generated using 7 trialsWarning: 'trials' should be <= 8 for this object. Predictions generated using 8 trialsWarning: 'trials' should be <= 9 for this object. Predictions generated using 9 trialsWarning: 'trials' should be <= 8 for this object. Predictions generated using 8 trialsWarning: 'trials' should be <= 5 for this object. Predictions generated using 5 trialsWarning: 'trials' should be <= 6 for this object. Predictions generated using 6 trialsWarning: 'trials' should be <= 3 for this object. Predictions generated using 3 trialsWarning: 'trials' should be <= 8 for this object. Predictions generated using 8 trialsWarning: 'trials' should be <= 9 for this object. Predictions generated using 9 trialsWarning: 'trials' should be <= 8 for this object. Predictions generated using 8 trials
c5model <- C5.0(salary_in_usd ~ .,
                       data = balanced_dataset,
                       trials = infoGain10$bestTune$trials, 
                       rules = FALSE,
                       control = C5.0Control(winnow = infoGain10$bestTune$winnow))
plot(c5model)

from the tree, the “experince level” attribute was the first selected splitting attribute meaning that it has the highest information gain among all attributes

Confusion matrix of 10 folds using Information gain

The following confusion Matrix will show the performance of the classifier using the predicted class labels and the actual class labels of our dataset

infoGain10cm= caret::confusionMatrix(infoGain10$pred$obs, infoGain10$pred$pred)

infoGain10cm
Confusion Matrix and Statistics

           Reference
Prediction  High Low Medium Very_High Very_Low
  High        90  19     40        20        5
  Low         24 127     42        10       67
  Medium      51  58     96        19       13
  Very_High   17   3     15       208        9
  Very_Low     3  43      4        10      204

Overall Statistics
                                              
               Accuracy : 0.6057              
                 95% CI : (0.5773, 0.6335)    
    No Information Rate : 0.249               
    P-Value [Acc > NIR] : < 0.0000000000000002
                                              
                  Kappa : 0.5046              
                                              
 Mcnemar's Test P-Value : 0.03427             

Statistics by Class:

                     Class: High Class: Low Class: Medium
Sensitivity              0.48649     0.5080        0.4873
Specificity              0.91700     0.8490        0.8590
Pos Pred Value           0.51724     0.4704        0.4051
Neg Pred Value           0.90714     0.8673        0.8948
Prevalence               0.15455     0.2089        0.1646
Detection Rate           0.07519     0.1061        0.0802
Detection Prevalence     0.14536     0.2256        0.1980
Balanced Accuracy        0.70174     0.6785        0.6732
                     Class: Very_High Class: Very_Low
Sensitivity                    0.7790          0.6846
Specificity                    0.9527          0.9333
Pos Pred Value                 0.8254          0.7727
Neg Pred Value                 0.9376          0.8992
Prevalence                     0.2231          0.2490
Detection Rate                 0.1738          0.1704
Detection Prevalence           0.2105          0.2206
Balanced Accuracy              0.8659          0.8089

similar to the trees from the gini index and gain ratio, the classifier seem to have better performance when treating the “very high” class as the positive class

5 Folds

The tree of the information gain using 5 folds

set.seed(10)
ctrl <- trainControl(method = "cv", number = 5, returnResamp="all", savePredictions="final")


infoGain5 <- train(salary_in_usd ~ ., data = balanced_dataset, method = "C5.0",trControl = ctrl)
Warning: 'trials' should be <= 7 for this object. Predictions generated using 7 trialsWarning: 'trials' should be <= 9 for this object. Predictions generated using 9 trialsWarning: 'trials' should be <= 8 for this object. Predictions generated using 8 trialsWarning: 'trials' should be <= 9 for this object. Predictions generated using 9 trialsWarning: 'trials' should be <= 8 for this object. Predictions generated using 8 trialsWarning: 'trials' should be <= 7 for this object. Predictions generated using 7 trialsWarning: 'trials' should be <= 5 for this object. Predictions generated using 5 trialsWarning: 'trials' should be <= 9 for this object. Predictions generated using 9 trialsWarning: 'trials' should be <= 5 for this object. Predictions generated using 5 trialsWarning: 'trials' should be <= 6 for this object. Predictions generated using 6 trialsWarning: 'trials' should be <= 8 for this object. Predictions generated using 8 trialsWarning: 'trials' should be <= 9 for this object. Predictions generated using 9 trialsWarning: 'trials' should be <= 9 for this object. Predictions generated using 9 trialsWarning: 'trials' should be <= 8 for this object. Predictions generated using 8 trialsWarning: 'trials' should be <= 7 for this object. Predictions generated using 7 trialsWarning: 'trials' should be <= 8 for this object. Predictions generated using 8 trials
c5model <- C5.0(salary_in_usd ~ .,
                       data = balanced_dataset,
                       trials = infoGain5$bestTune$trials, 
                       rules = FALSE,
                       control = C5.0Control(winnow = infoGain5$bestTune$winnow))
plot(c5model)

the tree has similar behavior as the 10 folds information gain tree

Confusion matrix of 5 folds using Information gain

The following confusion Matrix will show the performance of the classifier using the predicted class labels and the actual class labels of our dataset

infoGain5cm = caret::confusionMatrix(infoGain5$pred$obs, infoGain5$pred$pred)

infoGain5cm
Confusion Matrix and Statistics

           Reference
Prediction  High Low Medium Very_High Very_Low
  High        85  20     47        16        6
  Low         25 129     38        10       68
  Medium      37  70     99        14       17
  Very_High   16   4     13       210        9
  Very_Low     2  37      9        13      203

Overall Statistics
                                               
               Accuracy : 0.6065               
                 95% CI : (0.5782, 0.6343)     
    No Information Rate : 0.2531               
    P-Value [Acc > NIR] : < 0.00000000000000022
                                               
                  Kappa : 0.5049               
                                               
 Mcnemar's Test P-Value : 0.001691             

Statistics by Class:

                     Class: High Class: Low Class: Medium
Sensitivity              0.51515     0.4962       0.48058
Specificity              0.91376     0.8495       0.86075
Pos Pred Value           0.48851     0.4778       0.41772
Neg Pred Value           0.92180     0.8587       0.88854
Prevalence               0.13784     0.2172       0.17210
Detection Rate           0.07101     0.1078       0.08271
Detection Prevalence     0.14536     0.2256       0.19799
Balanced Accuracy        0.71446     0.6728       0.67066
                     Class: Very_High Class: Very_Low
Sensitivity                    0.7985          0.6700
Specificity                    0.9550          0.9318
Pos Pred Value                 0.8333          0.7689
Neg Pred Value                 0.9439          0.8928
Prevalence                     0.2197          0.2531
Detection Rate                 0.1754          0.1696
Detection Prevalence           0.2105          0.2206
Balanced Accuracy              0.8768          0.8009

the classifier shows very close performance to the 10 folds information gain model

3 Folds

The tree of the information gain using 3 folds

set.seed(10)
ctrl <- trainControl(method = "cv", number = 3, returnResamp="all", savePredictions="final")


infoGain3 <- train(salary_in_usd ~ ., data = balanced_dataset, method = "C5.0",trControl = ctrl)
Warning: 'trials' should be <= 4 for this object. Predictions generated using 4 trialsWarning: 'trials' should be <= 5 for this object. Predictions generated using 5 trialsWarning: 'trials' should be <= 6 for this object. Predictions generated using 6 trialsWarning: 'trials' should be <= 5 for this object. Predictions generated using 5 trialsWarning: 'trials' should be <= 6 for this object. Predictions generated using 6 trialsWarning: 'trials' should be <= 7 for this object. Predictions generated using 7 trialsWarning: 'trials' should be <= 7 for this object. Predictions generated using 7 trialsWarning: 'trials' should be <= 7 for this object. Predictions generated using 7 trialsWarning: 'trials' should be <= 9 for this object. Predictions generated using 9 trialsWarning: 'trials' should be <= 7 for this object. Predictions generated using 7 trialsWarning: 'trials' should be <= 9 for this object. Predictions generated using 9 trialsWarning: 'trials' should be <= 6 for this object. Predictions generated using 6 trials
c5model <- C5.0(salary_in_usd ~ .,
                       data = balanced_dataset,
                       trials = infoGain3$bestTune$trials, 
                       rules = FALSE,
                       control = C5.0Control(winnow = infoGain3$bestTune$winnow))
plot(c5model)

the visisble parts of the tree seem to behave the same as the prevoius 2 fold sizes- 10 and 5.

Confusion matrix of 3 folds using Information gain

The following confusion Matrix will show the performance of the classifier using the predicted class labels and the actual class labels of our dataset

infoGain3cm = caret::confusionMatrix(infoGain3$pred$obs, infoGain3$pred$pred)

infoGain3cm
Confusion Matrix and Statistics

           Reference
Prediction  High Low Medium Very_High Very_Low
  High        85  25     40        19        5
  Low         19 137     37         5       72
  Medium      55  64     93        11       14
  Very_High   16   8     22       197        9
  Very_Low     4  43     11         9      197

Overall Statistics
                                              
               Accuracy : 0.5923              
                 95% CI : (0.5639, 0.6203)    
    No Information Rate : 0.2481              
    P-Value [Acc > NIR] : < 0.0000000000000002
                                              
                  Kappa : 0.4874              
                                              
 Mcnemar's Test P-Value : 0.01149             

Statistics by Class:

                     Class: High Class: Low Class: Medium
Sensitivity              0.47486     0.4946       0.45813
Specificity              0.91257     0.8554       0.85513
Pos Pred Value           0.48851     0.5074       0.39241
Neg Pred Value           0.90811     0.8490       0.88542
Prevalence               0.14954     0.2314       0.16959
Detection Rate           0.07101     0.1145       0.07769
Detection Prevalence     0.14536     0.2256       0.19799
Balanced Accuracy        0.69372     0.6750       0.65663
                     Class: Very_High Class: Very_Low
Sensitivity                    0.8174          0.6633
Specificity                    0.9425          0.9256
Pos Pred Value                 0.7817          0.7462
Neg Pred Value                 0.9534          0.8928
Prevalence                     0.2013          0.2481
Detection Rate                 0.1646          0.1646
Detection Prevalence           0.2105          0.2206
Balanced Accuracy              0.8799          0.7944

since the tree is essentially similar to the previous two information gain trees the results that this tree shows is very close in performance to them as well.

Analysis of the information gain classification

The observed structure of the three decision trees seems to be the same and it can be summarized as follows:

  1. Root Node - Experience Level: The initial attribute used for splitting the dataset at the root node is the “experience level.” This divides the tree into two main branches or subtrees:

    • Right Subtree: This comprises instances with Senior (SE) and Executive (EX) experience levels.
    • Left Subtree: This includes individuals with Entry (EN) and Mid (MI) experience levels.
  2. Within the right subtree: In the right sub tree if the experience level is 4(EX) the tree will be divided based on “Company location”

  3. Within the left subtree: In the left subtree it will divide the tree for both two experience levels 1(EN) and 2(MI) based on “employee residence” and when the “employee residence” is “North America” the tree will be further divided based on “salary currency” and when this attribute is equal to “USD” the division will be based on the “job title” attribute

The decision tree continues to select the most appropriate attributes for splitting at each node, progressively refining the decision process until it reaches the leaves, where final class labels are assigned to the instances.

Sensitivity, Accuracy, Specifity and precision of all 3,5 and 10 folds using Information gain
rbind("10 Folds"=macro(infoGain10cm), "5 Folds"=macro(infoGain5cm), "3 Folds"=macro(infoGain3cm)  ) 

Based on the provided sensitivity, specificity, precision, and accuracy values there isn’t a clear indication of the superiority of one fold over another for Information Gain model .we may need to consider additional factors or conduct further analysis to make a well-informed decision. as can be seen in the table the 10 folds has the best Specificity and Precision, meanwhile the 5 folds has the best Sensitivity and Accuracy.

Analasys of the three classification methods

The gain ratio evaluated with 10-fold cross-validation appears to has the best performance among all the decision tree models. This might be due to the fact that the gain ratio tends to favor unbalanced splits, where one partition is significantly smaller than others. In our dataset, if an attribute has a rare value, the gain ratio might prioritize splitting on this attribute despite the resulting unbalance.

Despite the superiority of the 10-fold, the performance metrics of all three fold sizes (3-fold, 5-fold, and 10-fold) using Gini index, gain ratio, and information gain are relatively similar, suggesting that all three measures are robust within the context of this dataset. A likely contributing factor to this performance consistency is the balanced distribution of class labels in the dataset. When classes are balanced, each splitting criterion is more or less equally likely to encounter informative splits, which serves to decrease performance variability across different splitting methods.

Clustering

Data clustering is a process to partition data into groups or clusters,it is an unsupervised learning process, which is excuted without knowing the class label of the training data. The data in the same group “cluster” are similar to one another and different from data in other clusters. And for this data mining task We will utilize k-means clustering.

1- prepreocessing

we will encode the rest of factor columns to transform them into numeric types before clustering, enabling meaningful distance calculations using kmeans and other formulas, and allowing for maximum flexibility in data processing and interpretation. we will also remove the class label from the dataset as clustering is an unsupervised learning process, and we will preserve this class label in an attribute for later use. lastly, we will scale all numeric attributes in the dataset so they will be standarized.


# view data

dataset3 <- dataset2
View(dataset3)

# Reserve the salary_in_usd (the class label) column in an attribute before removing it from the dataset for clustering

classLabel <- dataset3[, 5] 


# Remove the class lable from the dataset

dataset3 <- dataset3[, -5]

# encoding job_title variable

dataset3$job_title = factor(dataset3$job_title, levels=c("Analyst", "Architect", "Engineer", "Leadership", "Consultant/Specialist","Cyber Security","Others" ), labels=c(4,1,2,5,3,6,7))

# encoding salary_currency variable

dataset3$salary_currency = factor(dataset3$salary_currency, levels=c("USD","BRL","GBP","EUR","INR","CAD","CHF","DKK","SGD","AUD","SEK","MXN","ILS","PLN","NOK","IDR","NZD","HUF","ZAR","TWD","RUB"), labels=c(1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21))

# encoding employee_residence variable

dataset3$employee_residence = factor(dataset3$employee_residence, levels=c("North America","Latin America & Caribbean","Sub-Saharan Africa", "Europe & Central Asia","East Asia & Pacific","South Asia","Middle East & North Africa"), labels=c(1,2,3,4,5,6,7))

# encoding company_location variable

dataset3$company_location = factor(dataset3$company_location, levels=c("North America","Latin America & Caribbean","Sub-Saharan Africa", "Europe & Central Asia","East Asia & Pacific","South Asia","Middle East & North Africa", "AQ", "UM"), labels=c(1,2,3,4,5,6,7,8,9))


 
#Data types to be transformed into numeric types before clustering
#Transforming all non-numeric attributes to numeric type


dataset3$experience_level <- as.numeric(as.character(dataset3$experience_level))

dataset3$job_title <- as.numeric(as.character(dataset3$job_title))

dataset3$salary_currency <- as.numeric(as.character(dataset3$salary_currency))

dataset3$employee_residence <- as.numeric(as.character(dataset3$employee_residence))

dataset3$company_location <- as.numeric(as.character(dataset3$company_location))

dataset3$company_size <- as.numeric(as.character(dataset3$company_size))

# viwe the class of attributes to ensure they have transformed to numeric
sapply(dataset3, class)
         work_year   experience_level          job_title 
         "numeric"          "numeric"          "numeric" 
   salary_currency employee_residence       remote_ratio 
         "numeric"          "numeric"          "numeric" 
  company_location       company_size 
         "numeric"          "numeric" 
#scale all attributes in the dataset so they would be standardized 
dataset3 <- scale(dataset3)

2- K-means

After preprocessing the data, now we are ready to perform the clustering process, we will use the k-means clustering, it is a clustering method that aims to minimize the sum of squared distances between each data point and the centroid of its assigned cluster by iteratively updating cluster assignments and centroids.

3- Choosing number of clusters (k)

We will choose 3 different numbers to perform the k-means clustering on, one of the numbers should be relatevily large, the second should be in the middle and the last should be small. This way we will cover the possible outcomes and clustering results.

a- Silhouette method

Now we will apply Silhouette method to find the optimal number of clusters k, we will also plot a graph where x-axis represent the number of clusters and y-axis represent the average Silhouette coefficient


fviz_nbclust(dataset3, kmeans, method = "silhouette")+
  labs(subtitle = "Silhouette method")

as seen by the graph, the number of clusters k that maximizes the average Silhouette coefficient is 2, so we will use it for clustering.

b- Elbow method

This method determines the number of clusters according to the turning point in a curve, the curve is plotted using the total within-cluster sum of square (WSS) as in y-axis , and No. clusters in x-axis


fviz_nbclust(dataset3, kmeans, method = "wss") +
  geom_vline(xintercept = 4, linetype = 2)+
  labs(subtitle = "Elbow method")

As shown, the number of clusters k that represents the turning point in the curve is 4, so we will use it for clustering.

Lastly, we will use k=3 since it acheives the second highest average Silhouette coefficient, and since it’s in the middle between 2 and 4 it will strike a balance between having too few clusters (k=2), and having several clusters (k=4), Thus, this choice will have an acceptable acuuracy.

k-means clustering, visualization and evaluation

In this section, we will perform k-means clustering and visualize its result using three different k’s that have been chosen beforehand, then we will compute WSS and Bcubed preceision and recall and average silhouette for each cluster as methods of evaluating clustering results.

k=2


#Use seed to guarantee replicability of random processes
set.seed(8953)

# run k-means clustering to find 2 clusters
kmeans.result <- kmeans(dataset3, 2)

# print the clusterng result
kmeans.result
K-means clustering with 2 clusters of sizes 938, 308

Cluster means:
   work_year experience_level   job_title salary_currency
1  0.1123321       0.07766291 -0.05252131      -0.3315729
2 -0.3421024      -0.23651887  0.15995128       1.0097901
  employee_residence remote_ratio company_location company_size
1         -0.5368064   0.08024947       -0.5259331    0.0410645
2          1.6348196  -0.24439612        1.6017052   -0.1250601

Clustering vector:
   [1] 1 1 1 1 2 1 1 1 2 2 1 1 1 1 1 1 1 1 1 2 2 2 2 1 1 2 1 2 1 1 1 1 1 1
  [35] 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 1 1 1 1 1 1 2 1 2 1 1 2 2 1 1 1 1
  [69] 1 1 1 1 1 1 1 2 1 1 2 1 1 1 1 1 1 1 1 1 2 1 1 2 1 2 1 1 1 1 1 1 2 1
 [103] 1 1 1 1 1 1 1 1 1 1 1 2 2 1 1 1 1 1 1 2 2 2 2 2 1 1 1 1 1 1 1 1 1 2
 [137] 2 2 2 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 1 1 1 1 1 1 1 1 1 1 1 1 1 1
 [171] 1 1 1 1 1 1 1 2 2 2 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 1 1 1 1 1 1
 [205] 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
 [239] 1 1 1 1 1 1 2 2 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 1 1 1 1 1 1
 [273] 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
 [307] 1 1 1 1 1 1 1 1 1 1 1 1 2 2 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
 [341] 1 1 1 1 1 1 1 1 1 1 1 1 2 1 2 1 1 1 1 1 1 1 1 1 1 1 1 2 2 1 1 1 1 1
 [375] 1 1 1 1 1 2 2 1 1 2 1 1 1 2 2 1 1 2 2 1 1 1 1 1 1 2 2 1 1 1 1 1 1 1
 [409] 1 1 1 1 1 1 1 1 1 2 2 1 1 1 1 1 1 1 1 1 1 1 1 1 2 1 2 1 2 2 1 2 1 1
 [443] 1 1 2 1 1 2 1 1 1 1 1 1 1 1 1 2 1 1 1 1 1 2 1 1 1 1 1 1 1 1 1 2 2 2
 [477] 1 1 1 1 2 2 2 2 1 1 2 1 1 2 2 1 2 2 1 1 2 1 2 2 1 1 1 1 1 1 1 1 1 2
 [511] 1 2 1 2 1 1 1 2 1 1 2 1 1 1 2 1 1 1 1 1 2 2 2 2 1 1 1 1 1 1 1 1 1 1
 [545] 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 1 1 2 2 1 1 1 1 1 1
 [579] 1 1 1 1 1 1 1 1 2 1 1 1 2 2 1 2 1 1 2 2 1 1 1 1 1 1 1 1 1 1 1 1 1 2
 [613] 1 2 2 2 2 1 1 1 1 2 2 2 2 2 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2
 [647] 1 1 1 1 1 1 1 2 1 2 1 1 2 1 2 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
 [681] 1 1 1 1 1 2 2 1 1 2 1 1 1 1 1 2 1 2 1 1 1 1 1 2 2 2 1 1 1 1 1 1 1 1
 [715] 1 1 1 2 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 2 1 1 1 1 1 2 1 2 1 1 1 2
 [749] 1 1 1 1 1 1 2 2 2 2 2 1 1 1 1 2 2 1 1 2 2 1 1 1 1 1 1 1 1 1 1 1 2 1
 [783] 1 2 2 2 1 1 1 2 1 2 1 1 1 2 1 1 1 1 2 2 1 1 1 1 2 1 1 2 1 1 1 1 2 1
 [817] 1 1 1 1 2 1 1 2 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 1 1 1 1 1 1 1 1 1
 [851] 1 1 1 2 1 1 1 1 2 1 1 1 2 2 1 1 1 2 2 1 1 1 2 1 2 1 1 2 1 1 2 1 1 1
 [885] 1 1 1 1 1 1 1 2 1 1 1 1 1 1 1 1 1 1 2 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
 [919] 2 2 1 1 2 1 1 2 1 1 1 1 1 1 1 1 2 1 1 1 1 1 1 1 1 1 1 2 1 1 1 2 2 1
 [953] 2 2 2 2 1 2 2 1 2 1 1 1 1 1 1 1 1 1 2 1 1 1 1 1 2 1 1 1 1 1 2 2 1 2
 [987] 2 2 2 2 1 1 1 1 1 2 1 1 1 2
 [ reached getOption("max.print") -- omitted 246 entries ]

Within cluster sum of squares by cluster:
[1] 4608.187 2679.470
 (between_SS / total_SS =  26.8 %)

Available components:

[1] "cluster"      "centers"      "totss"        "withinss"    
[5] "tot.withinss" "betweenss"    "size"         "iter"        
[9] "ifault"      
# visualize clustering
fviz_cluster(kmeans.result, data = dataset3)



#average silhouette for each clusters
avg_sil <- silhouette(kmeans.result$cluster,dist(dataset3)) 
fviz_silhouette(avg_sil)


#Within-cluster sum of squares wss 
wss <- kmeans.result$tot.withinss
print(wss)
[1] 7287.657
#BCubed
kmeans_cluster <- c(kmeans.result$cluster)

ground_truth <- c(classLabel)

data <- data.frame(cluster = kmeans_cluster, label = ground_truth)


  bcubed <- function(data) {
  n <- nrow(data)
  total_precesion <- 0
  total_recall <- 0

for (i in 1:n) {
  cluster <- data$cluster[i]
  label <- data$label[i]
    
# Number of objects in the same category and cluster
intersection <- sum(data$label[data$cluster == cluster] == label)
    
# Number of objects that are in the same cluster
total_same_cluster <- sum(data$cluster == cluster)
    
# Number of objects that have the same category
total_same_category <- sum(data$label == label)
    

total_precesion <- total_precesion + intersection /total_same_cluster
total_recall <- total_recall + intersection / total_same_category
  }

  # compute avg precision and recall
  precision <- total_precesion / n
  recall <- total_recall / n

  return(list(precision = precision, recall = recall)) }


# compute BCubed precision and recall
metrics <- bcubed(data)


precision <- metrics$precision
recall <- metrics$recall

# Print results
cat("BCubed Precision is:", precision, "\n")
BCubed Precision is: 0.2814606 
cat("BCubed Recall is:", recall, "\n")
BCubed Recall is: 0.7072104 

we can conclude from the graph and the results that the k=2 is the optimal k, since there is no overlapping between the two clusters, the data in a cluster are close “similar” to each other and dissimilar to data in the other cluster. Also, the recall is relatively high (0.71) and is the highest among the k’s chosen, the Precision is low (0.28) which could be duo to presence of outliers or sensitivity to Initial Centroid. We can also note that the WSS is 26.8%, which is an excellent percentage indicating a good compactness of clusters, and that objects in a cluster are similar to one another. Lastly, the average silhouette width is 0.34 which is considered high reflecting high intra-cluster similarity.

k=3


#Use seed to guarantee replicability of random processes
set.seed(8953)

# run k-means clustering to find 3 clusters
kmeans.result <- kmeans(dataset3, 3)

# print the clusterng result
kmeans.result
K-means clustering with 3 clusters of sizes 475, 302, 469

Cluster means:
   work_year experience_level    job_title salary_currency
1 -0.3742571       -0.4628902 -0.004526137      -0.3360410
2 -0.3447469       -0.2440022  0.169029312       1.0307121
3  0.6010356        0.6259307 -0.104257862      -0.3233595
  employee_residence remote_ratio company_location company_size
1         -0.5399390   -0.4866784       -0.5233822    0.2231005
2          1.6375401   -0.2426204        1.6311505   -0.1312858
3         -0.5076036    0.6491335       -0.5202578   -0.1414167

Clustering vector:
   [1] 1 1 1 1 2 3 3 3 2 2 1 3 3 1 3 3 3 1 1 2 2 2 2 1 1 2 3 2 3 3 1 3 1 1
  [35] 1 1 3 3 3 3 1 3 3 1 3 3 1 3 3 3 2 1 1 1 3 1 1 2 3 2 3 3 2 2 1 3 3 3
  [69] 3 3 3 3 3 3 3 2 1 1 2 3 3 3 3 3 3 3 3 3 2 3 3 2 1 2 3 3 3 3 3 3 2 1
 [103] 3 3 3 3 3 3 3 1 3 3 3 2 2 1 1 1 1 3 1 2 2 2 2 2 3 3 3 3 3 3 1 3 3 2
 [137] 2 2 2 3 3 1 1 3 3 3 1 1 3 3 3 3 3 3 3 2 3 3 1 1 1 1 3 3 3 1 1 3 3 3
 [171] 3 3 3 1 1 3 3 2 3 2 3 3 3 3 3 3 3 3 3 3 3 3 1 1 2 2 2 2 3 3 3 3 1 1
 [205] 3 3 3 3 3 3 3 3 3 3 1 1 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 1 1 1 1 1 1
 [239] 3 3 1 1 1 1 2 2 3 3 3 3 3 3 3 3 3 3 1 1 1 1 1 1 3 3 2 2 3 3 3 3 3 3
 [273] 1 1 3 3 3 3 3 3 3 3 3 3 1 1 3 3 3 3 3 3 3 3 3 3 1 1 1 1 1 1 3 3 3 3
 [307] 3 3 1 1 3 3 3 3 3 3 3 3 2 2 3 3 1 1 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3
 [341] 3 3 3 3 3 3 1 1 1 1 3 3 2 3 2 1 3 3 3 3 3 3 3 3 1 1 1 2 2 3 3 3 1 3
 [375] 3 3 1 3 3 2 2 1 3 2 3 3 3 2 2 3 3 2 2 1 3 1 3 3 3 2 2 3 1 1 3 1 3 3
 [409] 3 3 1 3 3 1 3 1 1 2 2 3 3 3 1 3 3 3 3 1 3 1 3 1 2 3 2 1 2 2 3 2 3 3
 [443] 3 1 2 1 3 2 3 1 1 1 1 3 3 1 1 2 1 3 3 3 3 2 1 3 1 3 1 3 1 1 1 2 2 2
 [477] 1 1 1 3 2 2 2 1 1 3 2 3 1 2 2 3 2 2 3 1 2 3 2 2 3 1 1 1 1 3 3 3 3 2
 [511] 1 2 3 2 3 1 3 2 1 3 2 3 3 3 2 1 3 1 3 3 2 2 2 2 3 3 3 3 1 3 3 1 1 3
 [545] 1 3 3 3 3 1 3 3 3 3 3 3 3 3 3 1 3 3 1 3 3 3 3 2 3 1 2 2 3 3 1 1 1 3
 [579] 3 3 3 1 1 1 1 1 2 3 1 1 2 2 1 2 3 3 2 2 1 3 3 3 3 1 3 3 3 3 1 1 3 3
 [613] 3 2 2 2 2 3 3 3 3 2 2 2 2 2 1 1 3 3 3 3 3 3 3 3 3 3 1 3 3 3 3 3 3 2
 [647] 3 1 1 3 3 3 3 2 1 2 3 1 2 1 2 3 3 3 1 1 3 3 3 1 3 1 3 1 1 3 1 1 1 1
 [681] 3 1 3 1 1 2 2 1 1 2 1 1 1 1 1 2 1 2 1 1 3 3 3 2 2 2 1 1 1 1 1 1 1 1
 [715] 1 1 3 2 3 1 1 1 3 1 1 3 1 1 1 1 1 1 3 1 2 2 1 3 1 1 3 2 1 2 1 1 1 2
 [749] 3 1 1 1 1 1 2 2 2 2 2 3 3 3 1 2 2 1 1 2 2 3 1 3 1 3 3 3 1 1 3 1 2 1
 [783] 1 2 2 2 1 1 1 2 1 2 1 1 1 2 1 1 1 1 2 2 1 1 1 1 2 3 1 2 3 1 1 3 2 1
 [817] 1 1 1 1 2 1 1 2 1 1 1 1 1 1 1 3 1 1 1 1 1 3 3 1 2 1 1 1 1 1 1 3 1 1
 [851] 1 3 3 2 3 1 1 1 2 1 1 1 1 2 1 3 1 2 2 1 1 1 2 1 2 1 3 2 1 1 2 1 3 1
 [885] 1 1 1 1 1 1 1 2 1 3 1 3 1 1 1 3 1 1 2 3 3 1 1 1 1 1 1 1 1 1 3 3 3 1
 [919] 2 2 3 1 2 1 3 2 3 1 1 1 1 1 1 1 2 1 3 1 1 1 3 1 3 1 1 2 1 1 3 2 2 1
 [953] 2 2 2 2 1 2 2 1 2 1 1 1 1 1 1 1 1 1 2 1 3 3 1 1 2 1 1 1 1 1 2 2 1 2
 [987] 2 2 2 2 1 1 1 1 1 2 1 1 1 2
 [ reached getOption("max.print") -- omitted 246 entries ]

Within cluster sum of squares by cluster:
[1] 2262.211 2615.908 1573.419
 (between_SS / total_SS =  35.2 %)

Available components:

[1] "cluster"      "centers"      "totss"        "withinss"    
[5] "tot.withinss" "betweenss"    "size"         "iter"        
[9] "ifault"      
# visualize clustering
fviz_cluster(kmeans.result, data = dataset3)


#average silhouette for each clusters
avg_sil <- silhouette(kmeans.result$cluster,dist(dataset3)) 
fviz_silhouette(avg_sil)


#Within-cluster sum of squares wss 
wss <- kmeans.result$tot.withinss
print(wss)
[1] 6451.538
#BCubed 
kmeans_cluster <- c(kmeans.result$cluster)

ground_truth <- c(classLabel)

data <- data.frame(cluster = kmeans_cluster, label = ground_truth)

  bcubed <- function(data) {
  n <- nrow(data)
  total_precesion <- 0
  total_recall <- 0

  for (i in 1:n) {
    cluster <- data$cluster[i]
    label <- data$label[i]
    
# Number of objects in the same category and cluster
intersection <- sum(data$label[data$cluster == cluster] == label)
    
# Number of objects that are in the same cluster
total_same_cluster <- sum(data$cluster == cluster)
    
# Number of objects that have the same category
total_same_category <- sum(data$label == label)
    

total_precesion <- total_precesion + intersection /total_same_cluster
total_recall <- total_recall + intersection / total_same_category
  }

  # compute avg precision and recall
  precision <- total_precesion / n
  recall <- total_recall / n

  return(list(precision = precision, recall = recall))
}

# compute BCubed precision and recall
metrics <- bcubed(data)


precision <- metrics$precision
recall <- metrics$recall

# Print results
cat("BCubed Precision is:", precision, "\n")
BCubed Precision is: 0.3059101 
cat("BCubed Recall is:", recall, "\n")
BCubed Recall is: 0.4537324 

we can conclude from the graph and the results where k=3 is that the performance is good but worse than k=2, because there is overlapping between clusters. In addition, the recall is relatively high (0.68), However, the Precision is low (0.28) which could be duo to presence of outliers or sensitivity to Initial Centroid. We can also note that the WSS is 32%, which is a good percentage indicating an intermidiate compactness of clusters, and that objects in a cluster are to some extent similar to one another. Lastly, the average silhouette width is 0.31 which is good reflecting a good intra-cluster similarity.

k=4

#Use seed to guarantee replicability of random processes
set.seed(8953)

# run k-means clustering to find 4 clusters
kmeans.result <- kmeans(dataset3, 4)

# print the clusterng result
kmeans.result
K-means clustering with 4 clusters of sizes 291, 259, 48, 648

Cluster means:
   work_year experience_level   job_title salary_currency
1 -0.2065990      -0.26057796 -0.07392709      -0.3742177
2 -0.2746520      -0.26852805  0.08713240       0.4713219
3 -0.7039739      -0.05919153  0.57502683       3.9452389
4  0.2547005       0.22873170 -0.04422191      -0.3125717
  employee_residence remote_ratio company_location company_size
1         -0.5494700   -1.2790118       -0.5320764   0.03437756
2          1.6082647   -0.1776153        1.5917788  -0.13081536
3          1.7560475   -0.5190540        1.7138693  -0.07831572
4         -0.5261344    0.6838108       -0.5242318   0.04264886

Clustering vector:
   [1] 1 1 1 1 2 4 4 4 2 2 1 4 4 1 4 4 4 4 4 2 2 2 2 1 1 2 4 2 4 4 4 4 1 1
  [35] 4 1 4 4 4 4 1 4 4 1 4 4 1 4 4 4 2 1 1 1 1 1 1 3 4 2 4 4 2 2 1 4 4 4
  [69] 4 4 4 4 4 4 4 2 4 4 2 4 4 4 4 4 4 4 4 4 3 4 4 3 1 2 4 4 4 4 4 4 2 4
 [103] 4 4 4 4 4 4 4 4 4 4 4 2 2 4 4 4 4 4 4 2 2 2 2 2 1 4 4 4 4 4 4 4 4 2
 [137] 2 2 2 4 4 1 1 4 4 4 4 1 4 4 4 4 4 4 4 2 4 4 4 4 1 1 4 4 4 1 1 4 4 4
 [171] 4 4 4 1 1 4 4 2 2 2 4 4 4 4 4 4 4 4 4 4 4 4 1 1 2 2 2 2 4 4 4 4 1 1
 [205] 4 4 4 4 4 4 4 4 4 4 1 1 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 1 1 1 1 1 1
 [239] 4 4 1 1 1 1 2 2 4 4 4 4 4 4 4 4 4 4 1 1 1 1 1 1 4 4 2 2 4 4 4 4 4 4
 [273] 1 1 4 4 4 4 4 4 4 4 4 4 1 1 4 4 4 4 4 4 4 4 4 4 1 1 1 1 1 1 4 4 4 4
 [307] 4 4 1 1 4 4 4 4 4 4 4 4 2 2 4 4 1 1 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4
 [341] 4 4 4 4 4 4 1 1 1 1 4 4 2 4 2 4 4 4 4 4 4 4 4 4 4 1 1 2 2 4 4 4 1 4
 [375] 4 4 4 4 4 2 2 1 4 2 4 4 4 2 2 4 4 2 2 4 4 1 4 4 4 2 2 4 1 1 4 1 4 4
 [409] 4 4 1 4 4 1 4 4 4 2 2 4 4 4 1 4 4 4 4 1 4 1 4 1 2 4 2 1 2 2 4 2 4 4
 [443] 4 1 2 1 4 2 4 1 1 1 1 4 4 1 1 2 4 4 4 4 4 3 4 4 4 4 4 4 1 1 1 2 2 3
 [477] 1 1 1 4 2 2 2 1 4 4 2 4 1 2 2 4 3 2 4 1 3 4 2 2 4 4 4 1 4 4 4 4 4 2
 [511] 4 2 4 2 4 4 4 2 4 4 2 4 4 4 2 1 4 4 4 4 2 2 3 2 4 4 4 4 1 4 4 4 4 4
 [545] 1 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 1 4 4 4 2 4 4 2 2 4 4 1 1 1 4
 [579] 4 4 4 4 1 1 4 1 2 4 4 1 2 2 4 2 4 4 3 2 4 4 4 4 4 4 4 4 4 4 4 1 4 2
 [613] 4 2 2 3 2 1 4 4 4 2 2 2 2 2 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 3
 [647] 4 1 1 4 4 4 4 2 4 2 4 4 2 1 2 4 4 4 1 4 4 4 4 4 4 4 4 1 1 4 1 4 1 4
 [681] 4 1 4 1 1 2 3 1 1 2 4 1 4 1 4 2 1 2 1 4 4 4 4 2 2 3 1 1 1 4 1 1 4 4
 [715] 1 4 4 2 4 4 1 1 4 1 1 4 1 4 4 4 1 4 4 2 2 2 4 4 4 4 4 2 4 2 1 4 1 2
 [749] 4 1 1 4 4 1 2 2 2 2 2 4 4 4 1 2 2 4 1 2 2 4 4 4 1 4 4 4 1 1 4 4 2 1
 [783] 1 2 2 2 4 1 1 3 1 3 1 1 1 3 1 4 1 4 2 2 1 4 1 1 3 4 1 2 4 4 4 4 2 4
 [817] 1 4 4 4 2 1 4 2 4 4 2 1 1 1 1 4 4 4 1 1 1 4 4 4 3 1 4 1 1 1 4 4 1 4
 [851] 4 4 4 3 4 1 1 1 2 1 1 1 2 2 4 4 1 3 3 4 1 1 2 1 2 4 4 2 1 1 2 1 4 4
 [885] 4 1 1 1 1 4 4 2 1 4 4 4 4 4 1 4 1 4 2 4 4 4 4 1 1 4 1 4 4 1 4 4 4 4
 [919] 2 2 4 1 2 1 4 3 4 4 1 1 4 4 4 4 2 4 4 1 4 4 4 4 4 1 1 2 1 4 4 2 2 1
 [953] 2 2 3 3 4 2 3 4 2 4 4 1 1 1 4 1 1 1 2 1 4 4 1 4 2 1 1 4 1 1 2 2 4 3
 [987] 3 2 2 3 1 1 1 1 1 2 4 4 4 2
 [ reached getOption("max.print") -- omitted 246 entries ]

Within cluster sum of squares by cluster:
[1] 1195.7362 1730.4906  422.4286 2562.3908
 (between_SS / total_SS =  40.7 %)

Available components:

[1] "cluster"      "centers"      "totss"        "withinss"    
[5] "tot.withinss" "betweenss"    "size"         "iter"        
[9] "ifault"      
# visualize clustering
fviz_cluster(kmeans.result, data = dataset3)


#average silhouette for each clusters
avg_sil <- silhouette(kmeans.result$cluster,dist(dataset3)) 
fviz_silhouette(avg_sil)


#Within-cluster sum of squares wss 
wss <- kmeans.result$tot.withinss
print(wss)
[1] 5911.046
#BCubed
kmeans_cluster <- c(kmeans.result$cluster)

ground_truth <- c(classLabel)

data <- data.frame(cluster = kmeans_cluster, label = ground_truth)


  bcubed <- function(data) {
  n <- nrow(data)
  total_precesion <- 0
  total_recall <- 0

  for (i in 1:n) {
    cluster <- data$cluster[i]
    label <- data$label[i]
    
# Number of objects in the same category and cluster
intersection <- sum(data$label[data$cluster == cluster] == label)
    
# Number of objects in the same  cluster
total_same_cluster <- sum(data$cluster == cluster)
    
# Number of objects that have the same category
total_same_category <- sum(data$label == label)
    
# Calculate precision and recall for the current item and add them to the sums
total_precesion <- total_precesion + intersection /total_same_cluster
total_recall <- total_recall + intersection / total_same_category
  }

  # Compute avg precision and recall
  precision <- total_precesion / n
  recall <- total_recall / n

  return(list(precision = precision, recall = recall))
}

# compute BCubed precision and recall
metrics <- bcubed(data)


precision <- metrics$precision
recall <- metrics$recall

# Print results
cat("BCubed Precision is:", precision, "\n")
BCubed Precision is: 0.2886446 
cat("BCubed Recall is:", recall, "\n")
BCubed Recall is: 0.4398836 

we can conclude from the graph and the results where k=4 is that the performance is worse than k=2 and k=3, because there is a noticeable overlapping between clusters. Also, the clusers’ space is pretty wide which results in a large distance between objects in the same cluster. In addition, the recall is relatively low (0.40) which might be a result of the overlapping and large distances between data objects. Furthermore, the Precision is low (0.28) which could be duo to presence of outliers or sensitivity to Initial Centroid. We can also note that the WSS is 40.7%, which is an acceptable percentage indicating a lower compactness of clusters. Lastly, the average silhouette width is 0.2 which is low reflecting high inter-cluster similarity.

LS0tDQp0aXRsZTogIkN5YmVyc2VjdXJpdHkgc2FsYXJpZXMiDQpvdXRwdXQ6IGh0bWxfbm90ZWJvb2sNCi0tLQ0KDQpgYGB7cn0NCg0Ka25pdHI6Om9wdHNfY2h1bmskc2V0KHdhcm5pbmc9RkFMU0UpDQoNCmBgYA0KDQojIyMgTmVlZGVkIGxpYnJhcmllcw0KDQpgYGB7cn0NCmxpYnJhcnkoZHBseXIpDQpsaWJyYXJ5KGNvdW50cnljb2RlKQ0KbGlicmFyeShvdXRsaWVycykNCmxpYnJhcnkoY2FyZXQpDQpsaWJyYXJ5KGNsdXN0ZXIpDQpsaWJyYXJ5KGZhY3RvZXh0cmEpDQpsaWJyYXJ5KE5iQ2x1c3QpDQpsaWJyYXJ5KCJETXdSIikNCmxpYnJhcnkoIlJXZWthIikNCmxpYnJhcnkoIkM1MCIpDQpsaWJyYXJ5KCJycGFydCIpDQpsaWJyYXJ5KCJ0aGVtaXMiKQ0KbGlicmFyeShyYXR0bGUpDQpsaWJyYXJ5KHJwYXJ0LnBsb3QpDQpsaWJyYXJ5KFJDb2xvckJyZXdlcikNCmBgYA0KDQojIHBoYXNlIDENCg0KIyMjIFByb2JsZW0gc3RhdGVtZW50DQoNClByZWRpY3Rpb24gb2YgY3liZXIgc2VjdXJpdHkgZW1wbG95ZWVzJyBzYWxhcmllcyBiYXNlZCBvbiAxMSBhdHRyaWJ1dGVzICYgZ3JvdXBpbmcgZW1wbG95ZWVzIGJhc2VkIG9uIHNoYXJlZCBjaGFyYWN0ZXJpc3RpY3MuDQoNCjEud29ya195ZWFyDQoNCjIuZXhwZXJpZW5jZV9sZXZlbA0KDQozLmVtcGxveW1lbnRfdHlwZQ0KDQo0LmpvYl90aXRsZQ0KDQo1LnNhbGFyeQ0KDQo2LnNhbGFyeV9jdXJyZW5jeQ0KDQo3LnNhbGFyeV9pbl91c2QNCg0KOC5lbXBsb3llZV9yZXNpZGVuY2UNCg0KOS5yZW1vdGVfcmF0aW8NCg0KMTAuY29tcGFueV9sb2NhdGlvbg0KDQoxMS5jb21wYW55X3NpemUNCg0KIyMjIFByb2JsZW0gZGVzY3JpcHRpb24NCg0KV2UgYXJlIGxpdmluZyBpbiB0aGUgImluZm9ybWF0aW9uIGFnZSIgb3IgcmF0aGVyIHRoZSAiZGF0YSBhZ2UiLCBtZWFuaW5nIHRoYXQgZXZlcnl0aGluZyBhcm91bmQgdXMgcmV2b2x2ZXMgYXJvdW5kIGRhdGEuIFRoZSBkYXRhIGhhcyBiZWNvbWUgb25lIG9mIHRoZSBtb3N0IHZhbHVhYmxlIGFzc2V0cyB0aGF0IGEgcGVyc29uIG9yIGFuIG9yZ2FuaXNhdGlvbiBjYW4gaGF2ZSwgc2luY2UgaXQgaGFzIGEgc2lnbmlmaWNhbnQgdmFsdWUsIGxvc2luZyBpdCB3aWxsIGxlYWQgdG8gc2lnbmlmaWNhbnQgZGFtYWdlcy4gVGh1cywgbW9zdCBvZiB0aGUgYXR0YWNrcyBub3dhZGF5cyBhcmUgZGlyZWN0ZWQgdG93YXJkIHRoZSBkYXRhLiBUbyBndWFyZCBhZ2FpbnN0IHN1Y2ggZGFtYWdlcywgb3JnYW5pc2F0aW9ucyBoYXZlIHJlYWxpc2VkIHRoZSBpbXBvcnRhbmNlIG9mIHByb3RlY3RpbmcgdGhlaXIgZGlnaXRhbCBhc3NldHMsIGxlYWRpbmcgdGhlbSB0byBoaXJlIGN5YmVyc2VjdXJpdHkgc3BlY2lhbGlzdHMuIFRoaXMgbWFkZSBjeWJlcnNlY3VyaXR5IGdhaW4gcG9wdWxhcml0eSBhbW9uZyBwZW9wbGUgc28gdGhlcmUncyBhIGdyb3dpbmcgdGVuZGVuY3kgdG8gc3R1ZHkgY3liZXJzZWN1cml0eS4gQ29uc2VxdWVudGx5IHRoaXMgcmVzdWx0ZWQgaW4gdGhlIGVtZXJnZW5jZSBvZiBwbGVudGlmdWwgcHJvZmVzc2lvbmFscyB3aXRoIHZhcmlvdXMgZXhwZXJpZW5jZSBsZXZlbHMgYW5kIHNraWxscyBpbiB0aGlzIGZpZWxkLiBBcyBhIHJlc3VsdCwgb3JnYW5pc2F0aW9ucyBtYXkgZmluZCBpdCBkaWZmaWN1bHQgdG8gZGVjaWRlIGEgc2FsYXJ5IGZvciBqb2IgY2FuZGlkYXRlcyBzb2xlbHkgYmFzZWQgb24gdGhlIENWLiBhbHNvLCBzaW5jZSB0aGUgYXR0YWNrcyBpbXByb3ZlIHJhcGlkbHksIG9yZ2FuaXNhdGlvbnMgbmVlZCB0byBoaXJlIG1vcmUgZW1wbG95ZWVzIGluIHRoZSBmYXIgZnV0dXJlIHRvIGRlZmVuZCBhZ2FpbnN0IHN1Y2ggYXR0YWNrcyBidXQgaXQncyBub3QgYW4gZWFzeSBtYXR0ZXIgdG8gcHJlZGljdCB0aGUgZnV0dXJlIHBheXJvbGwgd2hpY2ggbWF5IGhpbmRlcnMgc29tZSBvZiB0aGUgb3JnYW5pc2F0aW9uJ3MgcGxhbnMuIEFub3RoZXIgaXNzdWUgYXJpc2VzIHdoZW4gdGhlIGRlY2lzaW9uIG1ha2VycyBpbiB0aGUgb3JnYW5pc2F0aW9uIGFyZW4ndCBmdWxseSBhd2FyZSBvZiB0aGUgZGlmZmVyZW50IGdyb3VwcyBvZiBlbXBsb3llZXMgYW5kIHRoZWlyIGRpZmZlcmludCBuZWVkcy4gVGhlaXIgbGFjayBvZiBhd2FyZW5lc3MgZ2l2ZXMgYSBjaGFuY2UgZm9yIHRoZSBjb21wZXRpdG9yIG9yZ2FuaXNhdGlvbnMgdG8gYXR0cmFjdCB0aGVpciBlbXBsb3llZXMgdG8gdGhlbSBieSBvZmZlcmluZyBhIGJldHRlciBzYWxhcnkgYW5kIHByaXZpbGFnZXMgdGhhdCBtYXRjaCB0aGVpciBuZWVkcy4NCg0KIyMjIERhdGEgbWluaW5nIHRhc2sNCg0KUHJlZGljdGlvbiBvZiB0aGUgY3liZXIgc2VjdXJpdHkgZW1wbG95ZWVzJyBzYWxhcnkgY2F0ZWdvcmllcyAoVmVyeSBMb3csIExvdywgLCBIaWdoLCBWZXJ5IEhpZ2gpIHVzaW5nIGNsYXNzaWZpY2F0aW9uLCBhbmQgZGVzY3JpcHRpb24gb2YgZGF0YSBjaGFyYWN0ZXJpc3RpY3MgYW5kIGJlaGF2aW9yIGFuZCBncm91cGluZyBkYXRhIHVzaW5nIGNsdXN0ZXJpbmcgbWV0aG9kcy4NCg0KIyMjIEdvYWwNCg0KR2l2ZW4gdGhlIHByb2JsZW1zIHdlIGRpc2N1c3NlZCBhbmQgSW4gb3JkZXIgdG8gYmV0dGVyIHVuZGVyc3RhbmQgdGhpcyBmaWVsZCwgd2UgZGVjaWRlZCB0byBhbmFseXNlIGEgZGF0YXNldCBvZiAxMjQ3IGN5YmVyc2VjdXJpdHkgZW1wbG95ZWVzLCBjb250YWluaW5nIGluZm9ybWF0aW9uIHN1Y2ggYXMgc2FsYXJ5LCBqb2IgdGl0bGUsIGFuZCBleHBlcmllbmNlIGxldmVsLiBBbmFseXNpbmcgdGhpcyBkYXRhc2V0IGNhbiBwcm92aWRlIGluc2lnaHRmdWwgcHJlZGljdGlvbnMgcmVnYXJkaW5nIHRoZSBzYWxhcnkgcmFuZ2Ugb2YgYSBjeWJlcnNlY3VyaXR5IGVtcGxveWVlIGFuZCBkZXNjcmlwdGlvbiBvZiB0aGUgY3liZXJzZWN1cml0eSBtYXJrZXQgYmVoYXZpb3IgYnkgZ3JvdXBpbmcgdGhlIGRhdGEsIHdoaWNoIGNhbiBoZWxwIGluOg0KDQotICAgTWFya2V0IHNlZ21lbnRhdGlvbg0KLSAgIElkZW50aWZ5IHRyZW5kcw0KLSAgIFNwZWNpZnlpbmcgY29tbW9uIGNoYXJhY3RyZXN0aWNzIGFtb25nIGN5YmVyc2VjdXJpdHkgZW1wbG95ZWVzDQotICAgSWRlbnRpZnkgdGhlIG1haW4gY3liZXJzZWN1cml0eSBlbXBsb3llZSBncm91cHMgZm9yIGJldHRlciB1bmRlcnN0YW5kaW5nIHRoZWlyIG5lZWRzDQotICAgTWFraW5nIGJldHRlciBkZWNpc2lvbnMNCi0gICBNYWtpbmcgcmVjcnVpdG1lbnQgYW5kIGhpcmluZyBwcm9jZXNzIGVhc2llciBhbmQgbW9yZSBlZmZpY2llbnQNCi0gICBQcmVkaWN0aW5nIHRoZSBmdXR1cmUgcGF5cm9sbA0KLSAgIEluY3JlYXNpbmcgbG95YWx0eQ0KLSAgIEluY3JlYXNpbmcgdGhlIHNhdGlzZmFjdGlvbiByYXRlDQotICAgQWNoaWV2aW5nIGZhaXJuZXNzDQoNCiMjIFNvdXJjZSBvZiBkYXRhOg0KDQo8aHR0cHM6Ly93d3cua2FnZ2xlLmNvbS9kYXRhc2V0cy9kZWVwY29udHJhY3Rvci9jeWJlci1zZWN1cml0eS1zYWxhcmllcz4NCg0KIyMjIFJlYWRpbmcgYW5kIHZpZXdpbmcgZGF0YXNldA0KDQpgYGB7cn0NCmRhdGFzZXQ9IHJlYWQuY3N2KHVybCgiaHR0cHM6Ly9yYXcuZ2l0aHVidXNlcmNvbnRlbnQuY29tL1NhcmFoQWxoaW5kaS9ETV9wcm9qZWN0L21haW4vRGF0YSUyMFNldC9zYWxhcmllc19jeWJlci5jc3YiKSwgaGVhZGVyPVRSVUUpDQpWaWV3KGRhdGFzZXQpDQoNCmBgYA0KDQojIyMgT3JpZ2luYWwgZGF0YXNldA0KDQp3ZSB3aWxsIGtlZXAgYSBjb3B5IG9mIHRoZSBvcmlnaW5hbCBkYXRhc2V0IGJlZm9yZSBkYXRhIHByZXByb2Nlc3NpbmcgdG8gdXNlIGlmIG5lZWRlZCBhdCBhbnkgdGltZQ0KDQpgYGB7cn0NCm9yaWdpbmFsRGF0YXNldD0gZGF0YXNldA0KYGBgDQoNCiMjIEdlbmVyYWwgaW5mb3JtYXRpb24gYWJvdXQgdGhlIGRhdGFzZXQ6DQoNCk5vLiBvZiBhdHRyaWJ1dGVzOiAxMVwNClR5cGUgb2YgYXR0cmlidXRlczogT3JkaW5hbCAsIE5vbWluYWwsIGFuZCBOdW1lcmljXA0KTm8uIG9mIG9iamVjdHM6IDEyNDdcDQpDbGFzcyBsYWJlbDogc2FsYXJ5X2luX3VzZA0KDQpgYGB7cn0NCm5jb2woZGF0YXNldCkNCm5yb3coZGF0YXNldCkNCm5hbWVzKGRhdGFzZXQpDQpzdHIoZGF0YXNldCkNCmBgYA0KDQojIyMgQXR0cmlidXRlcycgZGVzY3JpcHRpb24gdGFibGUNCg0KKy0tLS0tLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tKw0KfCAqKkF0dHJpYnV0ZSBOYW1lKiogfCAqKkRlc2NyaXB0aW9uKiogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICB8ICoqRGF0YSBUeXBlKiogfCAqKlBvc3NpYmxlIHZhbHVlcyoqICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgfA0KKz09PT09PT09PT09PT09PT09PT09Kz09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT0rPT09PT09PT09PT09PT09Kz09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09Kw0KfCB3b3JrX3llYXIgICAgICAgICAgfCBUaGUgeWVhciBpbiB3aGljaCBzYWxhcnkgd2FzIHJlY29yZGVkICAgICAgICAgICAgICAgICAgICAgICB8IE51bWVyaWNhbCAgICAgfCAyMDIwIHRvIDIwMjIgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgfA0KKy0tLS0tLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tKw0KfCBleHBlcmllbmNlX2xldmVsICAgfCBFeHBlcnRpc2UgbGV2ZWwgb2YgdGhlIGVtcGxveWVlICAgICAgICAgICAgICAgICAgICAgICAgICAgICB8IE9yZGluYWwgICAgICAgfCBFbiAiRW50cnkgbGV2ZWwiXCAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgfA0KfCAgICAgICAgICAgICAgICAgICAgfCAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICB8ICAgICAgICAgICAgICAgfCBNSSAiTWlkIGxldmVsIlwgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgfA0KfCAgICAgICAgICAgICAgICAgICAgfCAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICB8ICAgICAgICAgICAgICAgfCBTRSAiU2VuaW9yIGxldmVsIlwgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgfA0KfCAgICAgICAgICAgICAgICAgICAgfCAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICB8ICAgICAgICAgICAgICAgfCBFWCAiRXhlY3V0aXZlIGxldmVsIiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgfA0KKy0tLS0tLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tKw0KfCBlbXBsb3ltZW50X3R5cGUgICAgfCBUaGUgbmF0dXJlIG9yIGNhdGVnb3J5IG9mIGVtcGxveWVlJ3MgZW5nYWdlbWVudCBpbiB0aGUgam9iICB8IE5vbWluYWwgICAgICAgfCBQVCAiUGFydCB0aW1lIlwgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgfA0KfCAgICAgICAgICAgICAgICAgICAgfCAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICB8ICAgICAgICAgICAgICAgfCBGVCAiRnVsbCB0aW1lIlwgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgfA0KfCAgICAgICAgICAgICAgICAgICAgfCAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICB8ICAgICAgICAgICAgICAgfCBDVCAiQ29udHJhY3RcICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgfA0KfCAgICAgICAgICAgICAgICAgICAgfCAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICB8ICAgICAgICAgICAgICAgfCBGTCJGcmVlbGFuY2VyIiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgfA0KKy0tLS0tLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tKw0KfCBqb2JfdGl0bGUgICAgICAgICAgfCBUaGUgcm9sZSB3b3JrZWQgaW4gZHVyaW5nIHRoZSB5ZWFyICAgICAgICAgICAgICAgICAgICAgICAgICB8IE5vbWluYWwgICAgICAgfCBEaWZmZXJlbnQgdGl0bGVzLiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgfA0KfCAgICAgICAgICAgICAgICAgICAgfCAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICB8ICAgICAgICAgICAgICAgfCAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgfA0KfCAgICAgICAgICAgICAgICAgICAgfCAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICB8ICAgICAgICAgICAgICAgfCBsaWtlIFNlY3VyaXR5IEFuYWx5c3QsIHNlY3VyaXR5IHJlc2VhcmNoZXIgICAgICAgICAgICAgICAgfA0KKy0tLS0tLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tKw0KfCBzYWxhcnkgICAgICAgICAgICAgfCBUaGUgdG90YWwgZ3Jvc3Mgc2FsYXJ5IGFtb3VudCBwYWlkICAgICAgICAgICAgICAgICAgICAgICAgICB8IE51bWVyaWNhbCAgICAgfCAxNzQwLTUwMDAxNTY2ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgfA0KKy0tLS0tLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tKw0KfCBzYWxhcnlfY3VycmVuY3kgICAgfCBUaGUgY3VycmVuY3kgb2YgdGhlIHNhbGFyeSBwYWlkIHRvIHRoZSBlbXBsb3llZSAgICAgICAgICAgICB8IE5vbWluYWwgICAgICAgfCBEaWZmZXJlbnQgY3VycmVuY2llcyBhY2NvcmRpbmcgdG8gSVNPIDQyMTcgY3VycmVuY3kgY29kZS4gfA0KfCAgICAgICAgICAgICAgICAgICAgfCAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICB8ICAgICAgICAgICAgICAgfCAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgfA0KfCAgICAgICAgICAgICAgICAgICAgfCAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICB8ICAgICAgICAgICAgICAgfCBsaWtlIERFLENBICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgfA0KKy0tLS0tLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tKw0KfCBzYWxhcnlfaW5fdXNkICAgICAgfCBUaGUgc2FsYXJ5IHBhaWQgaW4gVW5pdGVkIHN0YXRlcyBkb2xsYXIgICAgICAgICAgICAgICAgICAgICB8IE51bWVyaWNhbCAgICAgfCAyMDAwIHRvIDM2NTU5Ni40MCAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgfA0KKy0tLS0tLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tKw0KfCBlbXBsb3llZV9yZXNpZGVuY2UgfCBFbXBsb3llZSdzIHByaW1hcnkgY291bnRyeSBvZiByZXNpZGVuY2UgICAgICAgICAgICAgICAgICAgICB8IE5vbWluYWwgICAgICAgfCBEaWZmZXJlbnQgY291bnRyaWVzLiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgfA0KfCAgICAgICAgICAgICAgICAgICAgfCAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICB8ICAgICAgICAgICAgICAgfCAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgfA0KfCAgICAgICAgICAgICAgICAgICAgfCAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICB8ICAgICAgICAgICAgICAgfCBsaWtlIFVTLEFFICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgfA0KKy0tLS0tLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tKw0KfCByZW1vdGVfcmF0aW8gICAgICAgfCBQZXJjZW50YWdlIG9mIG9ubGluZSB3b3JrIGJ5IGVtcGxveWVlIGluIHRoZSBzcGVjaWZpZWQgeWVhciB8IE51bWVyaWNhbCAgICAgfCAwICJObyByZW1vdGUgd29yayJcICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgfA0KfCAgICAgICAgICAgICAgICAgICAgfCAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICB8ICAgICAgICAgICAgICAgfCA1MCAiUGFydGlhbGx5IHJlbW90ZSJcICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgfA0KfCAgICAgICAgICAgICAgICAgICAgfCAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICB8ICAgICAgICAgICAgICAgfCAxMDAgIkZ1bGx5IHJlbW90ZSIgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgfA0KKy0tLS0tLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tKw0KfCBjb21wYW55X2xvY2F0aW9uICAgfCBUaGUgY291bnRyeSBvZiB0aGUgZW1wbG95ZXIncyBtYWluIG9mZmljZSAgICAgICAgICAgICAgICAgICB8IE5vbWluYWwgICAgICAgfCBEaWZmZXJlbnQgY291bnRyaWVzLiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgfA0KfCAgICAgICAgICAgICAgICAgICAgfCAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICB8ICAgICAgICAgICAgICAgfCAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgfA0KfCAgICAgICAgICAgICAgICAgICAgfCAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICB8ICAgICAgICAgICAgICAgfCBsaWtlIEJSLEJXICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgfA0KKy0tLS0tLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tKw0KfCBjb21wYW55X3NpemUgICAgICAgfCBIb3cgYmlnL3NtYWxsIGlzIHRoZSBjb21wYW55ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICB8IE9yZGluYWwgICAgICAgfCBTICwgTSBvciBMICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgfA0KKy0tLS0tLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tKw0KDQojIHBoYXNlIDINCg0KIyMjIHNhbXBsZSBvZiAyMCBlbXBsb3llZXMgZnJvbSB0aGUgZGF0YXNldDoNCg0KdXNpbmcgc2FtcGxlX24odGFibGUsc2l6ZSkgZnVuY3Rpb24gYW5kIHVzaW5nIChzZXRfc2VlZCgpKQ0KDQpgYGB7cn0NCnNldC5zZWVkKDMwKQ0Kc2FtcGxlPXNhbXBsZV9uKGRhdGFzZXQsMjApDQpwcmludChzYW1wbGUpDQpgYGANCg0KIyMjIFNob3cgdGhlIG1pc3NpbmcgdmFsdWU6DQoNCmlmIGl0IGlzIEZBTFNFIGl0IG1lYW5zIG5vIG51bGwgdmFsdWUsaWYgaXQgaXMgVFJVRSB0aGVyZSBpcyBudWxsIHZhbHVlLiBJbiBvdXIgZGF0YXNldCB0aGVyZSBpcyBubyBudWxsIHZhbHVlcy4NCg0KYGBge3J9DQppcy5uYShkYXRhc2V0KQ0Kc3VtKGlzLm5hKGRhdGFzZXQpKQ0KYGBgDQoNCiMjIyBTaG93IHRoZSBNaW4uLDFzdCBRdS4sTWVkaWFuLE1lYW4gLDNyZCBRdS4sTWF4LiBmb3IgZWFjaCBudW1lcmljIGNvbHVtbg0KDQpUaGUgc3VtbWFyeSBzdGF0aXN0aWNzIGZvciB0aGUgZGF0YXNldCB2YXJpYWJsZXMgcHJvdmlkZSBpbnNpZ2h0cyBpbnRvIHRoZSBkaXN0cmlidXRpb24gb2YgZmVhdHVyZXMuDQp3ZSBjYW4gY29uY2x1ZGUgdGhlIGZvbGxvd2luZzoNCg0KSW4gd29ya195ZWFyOiBUaGUgZGF0YSBzcGFucyBmcm9tIHRoZSB5ZWFyIDIwMjAgdG8gMjAyMiB3aXRoIE1vc3QgZGF0YSBmYWxsaW5nIHdpdGhpbiB0aGUgeWVhcnMgMjAyMSBhbmQgMjAyMiwgYXMgaW5kaWNhdGVkIGJ5IGJvdGggdGhlIG1lZGlhbiBhbmQgbWVhbiBiZWluZyBjZW50ZXJlZCBhcm91bmQgMjAyMS4NCg0KSW4gc2FsYXJ5OiBTYWxhcmllcyB2YXJ5IHdpZGVseSB3aXRoIGEgbWluaW11bSBvZiAkMSw3NDAgYW5kIGEgbWF4aW11bSBvZiAkNTAwIG1pbGxpb24uIFRoZSBtZWRpYW4gaXMgJDEyMCwwMDAgd2hpY2ggaXMgYSBtaWQgdmFsdWUsIGJ1dCB0aGUgbWVhbiBpcyBub3RhYmx5IGhpZ2hlciBhdCAkNTYwLDg1MiB3aGljaCBtaWdodCBiZSBkdW8gdG8gZXh0cmVtZSB2YWx1ZXMgb3Igbm90YWJsZSBza2V3bmVzcy4NCg0KSW4gc2FsYXJ5X2luX3VzZDogVGhlIGRhdGEgaGFzIGEgbWVkaWFuIG9mICQxMTAsMDAwLCBhbmQgYSBtZWFuIG9mICQxMjAsMjc4LCBhbmQgdGhlIHNwcmVhZCBvZiBzYWxhcmllcyBpcyBvYnNlcnZhYmxlIGluIHRoZSBkaWZmZXJlbmNlIGJldHdlZW4gdGhlIG1lZGlhbiBhbmQgbWVhbi4NCg0KSW4gcmVtb3RlX3JhdGlvOiBJbmRpY2F0ZXMgdGhlIHBlcmNlbnRhZ2Ugb2YgcmVtb3RlIHdvcmsgcmFuZ2luZyBmcm9tIDAlIHRvIDEwMCUsIHdpdGggYSBtZWRpYW4gYW5kIDNyZCBxdWFydGlsZSBhdCAxMDAlLCBhbmQgYSBtZWFuIG9mIDcxLjQ5JSwgaW5kaWNhdGluZyBhIG5vdGFibGUgcHJlc2VuY2Ugb2YgcmVtb3RlIHdvcmsgaW4gdGhlIGRhdGFzZXQsIHN1Z2dlc3Rpbmcgc29tZSB2YXJpYWJpbGl0eS4gDQoNCmBgYHtyfQ0Kc3VtbWFyeShkYXRhc2V0JHdvcmtfeWVhcikNCnN1bW1hcnkoZGF0YXNldCRzYWxhcnkpDQpzdW1tYXJ5KGRhdGFzZXQkc2FsYXJ5X2luX3VzZCkNCnN1bW1hcnkoZGF0YXNldCRyZW1vdGVfcmF0aW8pDQpgYGANCg0KDQoNCiMjIyBTaG93IHRoZSB2YXJpYW5lIG9mIGVhY2ggbnVtZXJpYyBjb2x1bW4NCg0KdmFyaWFuY2UgaXMgdG8gdW5kZXJzdGFuZCB0aGUgc3ByZWFkIG9yIGRpc3BlcnNpb24gb2YgdGhlIHZhbHVlcyBpbiBlYWNoIGNvbHVtbi4gQSBoaWdoZXIgdmFyaWFuY2UgaW5kaWNhdGVzIHRoYXQgdGhlIHZhbHVlcyBhcmUgbW9yZSBzcHJlYWQgb3V0IGZyb20gdGhlIG1lYW4gYW5kIGluIG91ciBkYXRhc2V0IHRoZSBoaWdoZXN0IHZhcmllZCBhdHRyaWJ1dGUgaXMgc2FsYXJ5LCB3aGlsZSBhIGxvd2VyIHZhcmlhbmNlIGluZGljYXRlcyB0aGF0IHRoZSB2YWx1ZXMgYXJlIGNsb3NlciB0byB0aGUgbWVhbiB3aGljaCBpbiBvdXIgZGF0YXMgaXQgaXMgd29yayB5ZWFyIGF0dHJpYnV0ZS4NCg0KVmFyaWFuY2UgcmVzdWx0cyByZXZlYWwgdGhhdDoNCi13b3JrIHllYXJzIGFyZSB0byBzb21lIGV4dGVudCBjb25zaXN0ZW50IA0KLXNhbGFyaWVzIHNob3cgbm90YWJsZSB2YXJpYWJpbGl0eSBhbmQgcG9zc2libGUgb3V0bGllcnMNCi1zYWxhcmllcyBpbiBVU0QgaGF2ZSBhIHN0YWJsZSBkaXN0cmlidXRpb24gDQotcmVtb3RlIHdvcmsgcmF0aW8gaGF2ZSBtb2RlcmF0ZSB2YXJpYWJpbGl0eQ0KDQoNCmBgYHtyfQ0KdmFyKGRhdGFzZXQkd29ya195ZWFyKQ0KdmFyKGRhdGFzZXQkc2FsYXJ5KQ0KdmFyKGRhdGFzZXQkc2FsYXJ5X2luX3VzZCkNCnZhcihkYXRhc2V0JHJlbW90ZV9yYXRpbykNCmBgYA0KDQojIyMgVmlzdWFsaXphdGlvbiBvZiByZWxhdGlvbnNoaXAgYmV0d2VlbiBzb21lIHBhaXJzIG9mIGF0dHJpYnV0ZXM6DQoNCkhlcmUgd2UgdXNlZCBib3hwbG90IHRvIHNlZSB0aGUgZGlzdHJpYnV0aW9uIGJldHdlZW4gc2FsYXJ5X2luX3VzZCBhbmQgZXhwZXJpZW5jZV9sZXZlbCBXZSBvYnNlcnZlZCB0aGF0IHNhbGFyaWVzIHZhcnkgZGVwZW5kaW5nIG9uIHRoZSBsZXZlbCBvZiBleHBlcmllbmNlLHRoZXkgYXJlIHBvc2l0aXZlbHkgY29ycmVsYXRlZC4NCg0KYGBge3J9DQpib3hwbG90KHNhbGFyeV9pbl91c2QgfiBleHBlcmllbmNlX2xldmVsLCBkYXRhID0gZGF0YXNldCAsIHlheHQ9Im4iKQ0KbGFiZWxzPC0gcHJldHR5KGRhdGFzZXQkc2FsYXJ5X2luX3VzZCkNCmxhYmVsczwtIHNhcHBseShsYWJlbHMsIGZ1bmN0aW9uKHgpIGZvcm1hdCh4LCBzY2llbnRpZmljID0gRkFMU0UpKQ0KYXhpcyhzaWRlID0gMiwgYXQ9cHJldHR5KGRhdGFzZXQkc2FsYXJ5X2luX3VzZCksIGxhYmVscyA9IGxhYmVscyApDQpvcHRpb25zKHNjaXBlbiA9IDk5OSkNCmBgYA0KDQpIZXJlIHdlIHVzZWQgYm94cGxvdCB0byBzZWUgdGhlIGRpc3RyaWJ1dGlvbiBiZXR3ZWVuIHNhbGFyeV9pbl91c2QgYW5kIHdvcmtfeWVhciBXZSBvYnNlcnZlZCB0aGF0IDIwMjEgc2FsYXJpZXMgd2VyZSBjbG9zZSB0byBlYWNoIG90aGVyIGJ1dCBpbiAyMDIyIHRoZSBnYXAgYmV0d2VlbiB0aGVtIGdldHRpbmcgYmlnZ2VyLg0KDQpgYGB7cn0NCmJveHBsb3Qoc2FsYXJ5X2luX3VzZCB+IHdvcmtfeWVhciwgZGF0YSA9IGRhdGFzZXQgLCB5YXh0PSJuIikNCmxhYmVsczwtIHByZXR0eShkYXRhc2V0JHNhbGFyeV9pbl91c2QpDQpsYWJlbHM8LSBzYXBwbHkobGFiZWxzLCBmdW5jdGlvbih4KSBmb3JtYXQoeCwgc2NpZW50aWZpYyA9IEZBTFNFKSkNCmF4aXMoc2lkZSA9IDIsIGF0PXByZXR0eShkYXRhc2V0JHNhbGFyeV9pbl91c2QpLCBsYWJlbHMgPSBsYWJlbHMgKQ0Kb3B0aW9ucyhzY2lwZW4gPSA5OTkpDQpgYGANCg0KSGVyZSB3ZSB1c2VkIGJveHBsb3QgdG8gc2VlIHRoZSBkaXN0cmlidXRpb24gYmV0d2VlbiBzYWxhcnlfaW5fdXNkIGFuZCBlbXBsb3ltZW50X3R5cGUgV2Ugb2JzZXJ2ZWQgdGhhdCBGdWxsIFRpbWUgKEZUKSBvZmZlcnMgbW9yZSBzYWxhcnkgdGhhbiB0aGUgb3RoZXIgY2F0ZWdvcmllcy4NCg0KYGBge3J9DQpib3hwbG90KHNhbGFyeV9pbl91c2QgfiBlbXBsb3ltZW50X3R5cGUsIGRhdGEgPSBkYXRhc2V0ICwgeWF4dD0ibiIpDQpsYWJlbHM8LSBwcmV0dHkoZGF0YXNldCRzYWxhcnlfaW5fdXNkKQ0KbGFiZWxzPC0gc2FwcGx5KGxhYmVscywgZnVuY3Rpb24oeCkgZm9ybWF0KHgsIHNjaWVudGlmaWMgPSBGQUxTRSkpDQpheGlzKHNpZGUgPSAyLCBhdD1wcmV0dHkoZGF0YXNldCRzYWxhcnlfaW5fdXNkKSwgbGFiZWxzID0gbGFiZWxzICkNCm9wdGlvbnMoc2NpcGVuID0gOTk5KQ0KYGBgDQoNCkhlcmUgd2UgdXNlZCBib3hwbG90IHRvIHNlZSB0aGUgZGlzdHJpYnV0aW9uIGJldHdlZW4gc2FsYXJ5X2luX3VzZCBhbmQgY29tcGFueV9zaXplIFdlIG9ic2VydmVkIHRoYXQgdGhlIGxhcmdlciB0aGUgY29tcGFueSBpcyB0aGUgaGlnaGVyIHRoZSBzYWxhcnkgd2FzLg0KDQpgYGB7cn0NCmJveHBsb3Qoc2FsYXJ5X2luX3VzZCB+IGNvbXBhbnlfc2l6ZSwgZGF0YSA9IGRhdGFzZXQgLCB5YXh0PSJuIikNCmxhYmVsczwtIHByZXR0eShkYXRhc2V0JHNhbGFyeV9pbl91c2QpDQpsYWJlbHM8LSBzYXBwbHkobGFiZWxzLCBmdW5jdGlvbih4KSBmb3JtYXQoeCwgc2NpZW50aWZpYyA9IEZBTFNFKSkNCmF4aXMoc2lkZSA9IDIsIGF0PXByZXR0eShkYXRhc2V0JHNhbGFyeV9pbl91c2QpLCBsYWJlbHMgPSBsYWJlbHMgKQ0Kb3B0aW9ucyhzY2lwZW4gPSA5OTkpIA0KYGBgDQoNCiMjIERhdGEgUmVkdWN0aW9uDQoNCiMjIyBEaW1lbnNpb25hbGl0eSBSZWR1Y3Rpb24NCg0KVGhlICJzYWxhcnkiIGNvbHVtbiBnaXZlcyB0aGUgc2FtZSBpbmZvcm1hdGlvbiBhcyAic2FsYXJ5X2luX3VzZCIgaXQncyBqdXN0IGEgbWF0dGVyIG9mIGN1cnJlbmN5IGV4Y2hhbmdlLCBhbmQgd2Ugd2lsbCBldmVudHVhbGx5IHRyYW5zZm9ybSBhbGwgdGhlIHZhbHVlcyBpbiAic2FsYXJ5IiBjb2x1bW4gdG8gb25lIGNvbW1vbiBjdXJyZW5jeSBzbyB3ZSBjYW4gcHJvcGVybHkgZGVhbCB3aXRoIHRoZW0uIFRvIGZ1cnRoZXIgY29uZmlybSB0aGF0IHRoZSB0d28gY29sdW1uIGFyZSByZWR1bmRhbnQsIHdlIHdpbGwgdXNlIHRoZSBsYXRlc3QgZXhjaGFuZ2UgcmF0ZSBmb3IgVVNEIHRvIHRoZSBkZXNpcmVkIGN1cnJlbmN5Lg0KDQp3ZSB3aWxsIHN0YXJ0IGJ5IGNyZWF0aW5nIGEgdGVtcG9yYXJ5IGNvbHVtbiBuYW1lZCAiY29udmVydGVkX3NhbGFyeSIgdG8gc2F2ZSB0aGUgc2FsYXJ5IHRoYXQgd2Ugd2lsbCBnZXQgYnkgdXNpbmcgdGhlIGV4Y2hhbmdlIHJhdGUgdG8gY29udmVydCB0aGUgc2FsYXJ5X2luX3VzZCB0byB0aGUgc2FsYXJ5IHdpdGggZGlmZmVyZW50IGN1cnJlbmNpZXMgdG8gY29tcGFyZSB3aXRoICJzYWxhcnkiIGNvbHVtbg0KDQpgYGB7cn0NCmNvbnZlcnRlZERhdGFzZXQ9ZGF0YXNldA0KDQoNCmNvbnZlcnRlZERhdGFzZXQkZXhjaGFuZ2VfcmF0ZSA9IGZhY3Rvcihjb252ZXJ0ZWREYXRhc2V0JHNhbGFyeV9jdXJyZW5jeSwgbGV2ZWxzPWMoIlVTRCIsIkJSTCIsIkdCUCIsIkVVUiIsIklOUiIsIkNBRCIsIkNIRiIsIkRLSyIsIlNHRCIsIkFVRCIsIlNFSyIsIk1YTiIsIklMUyIsIlBMTiIsIk5PSyIsIklEUiIsIk5aRCIsIkhVRiIsIlpBUiIsIlRXRCIsIlJVQiIpLCBsYWJlbHM9YygxLzEsMS8wLjIwLDEvMS4yMiwxLzEuMDYsMS8wLjAxMiwxLzAuNzQsMS8xLjEwLDEvMC4xNCwxLzAuNzMsMS8wLjY0LDEvMC4wOTAsMS8wLjA1NywxLzAuMjYsMS8wLjIzLDEvMC4wOTMsMS8wLjAwMDA2NSwxLzAuNjAsMS8wLjAwMjcsMS8wLjA1MywxLzAuMDMxLDEvMC4wMTApKQ0KY29udmVydGVkRGF0YXNldCRleGNoYW5nZV9yYXRlID0gYXMubnVtZXJpYyhhcy5jaGFyYWN0ZXIoY29udmVydGVkRGF0YXNldCRleGNoYW5nZV9yYXRlKSkNCmNvbnZlcnRlZERhdGFzZXQkY29udmVydGVkX3NhbGFyeSA9IGNvbnZlcnRlZERhdGFzZXQkc2FsYXJ5X2luX3VzZCpjb252ZXJ0ZWREYXRhc2V0JGV4Y2hhbmdlX3JhdGUNCg0KDQoNCnNldC5zZWVkKDEpDQpzYWxhcnlfc2FtcGxlIDwtIHNhbXBsZV9uKGNvbnZlcnRlZERhdGFzZXRbLGMoInNhbGFyeSIsImNvbnZlcnRlZF9zYWxhcnkiKV0sMTApDQoNCnByaW50KHNhbGFyeV9zYW1wbGUpDQpgYGANCg0KYXMgc2hvd24gaW4gdGhlIHNhbXBsZSwgdGhlIHR3byBjb2x1bW5zIGFyZSBhbG1vc3QgaWRlbnRpY2FsLiBUaGlzIGNhbiBiZSBwcm92ZWQgYnkgY29ycmVsYXRpb24gY29lZmZpY2llbnQgYXMgd2VsbC4NCg0KYGBge3J9DQpjb3JyZWxhdGlvbiA8LSBjb3IoY29udmVydGVkRGF0YXNldCRzYWxhcnkgLCBjb252ZXJ0ZWREYXRhc2V0JGNvbnZlcnRlZF9zYWxhcnkpDQpwcmludChjb3JyZWxhdGlvbikNCmBgYA0KDQpUaGUgY29ycmVsYXRpb24gaXMgc28gaGlnaCBidXQgaXQgaGFzbid0IHJlYWNoZWQgMTAwJSBwb3NzaWJseSBkdWUgdG8gcm91bmRpbmcgaW4gdGhlIGNhbGN1bGF0aW9ucyBhbmQgc2xpZ2h0IGRpZmZlcmVuY2VzIGluIHRoZSBleGNoYW5nZSByYXRlIG92ZXIgdGltZS4NCg0KVG8gbWFrZSB0aGUgbWluaW5nIHByb2Nlc3MgbW9yZSBlZmZpZWNlbnQgYW5kIGhhcyBhbiBpbXByb3ZlZCBxdWFsaXR5LCB3ZSBkZWNpZGVkIHRvIHJlbW92ZSB0aGUgInNhbGFyeSIgY29sdW1uLg0KDQpgYGB7cn0NCmRhdGFzZXQ8LWRhdGFzZXRbLC1jKDUpXQ0KYGBgDQoNCiMjIyBGaW5kIHRoZSBvdXRsaWVycyBhbmQgcmVtb3ZlIHRoZW06DQoNCldlIHdpbGwgc2hvdyBvdXRsaWVycyB3aXRoIGJveFBsb3RzIGFuZCB0aGVuIHJlbW92ZSB0aGVtLCB0byBtaW5pbWl6ZSBub2lzZSBhbmQgdG8gZ2V0IGJldHRlciBhbmFseXRpY2FsIHJlc3VsdHMgd2hlbiBhcHBseWluZyBkYXRhIG1pbmluZyB0ZWNobmlxdWVzLg0KDQpub3cgd2Ugc2hvdyAoc2FsYXJ5X2luX3VzZCkgYXR0cmlidXRlcycgb3V0bGllcnMuIHdlIGNhbiBzZWUgdGhhdCB0aGVyZSBhcmUgbWFueSBvdXRsaWVycyB3aXRoIGV4Y2VwdGlvbmFsbHkgaGlnaCB2YWx1ZXMsIHRodXMgd2Ugd2lsbCByZW1vdmUgdGhlbS4NCg0KYGBge3J9DQpib3hwbG90KGRhdGFzZXQkc2FsYXJ5X2luX3VzZCkNCg0KDQoNCk91dFNhbGFyeSA9IG91dGxpZXIoZGF0YXNldCRzYWxhcnlfaW5fdXNkLCBsb2dpY2FsID1UUlVFKQ0KRmluZF9vdXRsaWVyID0gd2hpY2goT3V0U2FsYXJ5ID09VFJVRSwgYXJyLmluZCA9IFRSVUUpDQpkYXRhc2V0PSBkYXRhc2V0Wy1GaW5kX291dGxpZXIsXQ0KDQpgYGANCg0Kbm93IHdlIHNob3cgKHJlbW90ZV9yYXRpbykgYXR0cmlidXRlcycgb3V0bGllcnMuIHdlIGNhbiBzZWUgdGhlcmUgYXJlbid0IG91dGxpZXJzIGluIHJlbW90ZV9yYXRpbywgdGh1cyB3ZSBkb24ndCBuZWVkIHRoZSBsYXN0IHN0ZXAgaS5lOiByZW1vdmluZyBvdXRsaWVycycgcm93cy4NCg0KYGBge3J9DQpib3hwbG90KGRhdGFzZXQkcmVtb3RlX3JhdGlvKQ0KDQpgYGANCg0Kbm93IHdlIHNob3cgKHdvcmtfeWVhcikgYXR0cmlidXRlcycgb3V0bGllcnMuIHdlIGNhbiBzZWUgdGhlcmUgYXJlbid0IG91dGxpZXJzIGluIHdvcmtfeWVhciwgdGh1cyB3ZSBkb24ndCBuZWVkIHRoZSBsYXN0IHN0ZXAgaS5lOiByZW1vdmluZyBvdXRsaWVycycgcm93cy4NCg0KYGBge3J9DQpib3hwbG90KGRhdGFzZXQkd29ya195ZWFyKQ0KDQpgYGANCg0KIyMjIENvbmNlcHQgaGllcmFyY2h5IGdlbmVyYXRpb24gZm9yIG5vbWluYWwgZGF0YQ0KDQp0aGUgY29sdW1ucyAiY29tcGFueV9sb2NhdGlvbiIgYW5kICJlbXBsb3llZV9yZXNpZGVuY2UiIGhhdmUgdGhlIG5hbWUgb2YgY291bnRyaWVzIGZvciB0aGUgY29tcGFueSBhbmQgZW1wbG95ZWUgcmVzcGVjdGl2ZWx5LiBBbmQgdGhlc2UgYXR0cmlidXRlcyBjYW4gYmUgZ2VuZXJhbGl6ZWQgdG8gaGlnaGVyLWxldmVsIGNvbmNlcHQgdGhhdCBpcyByZWdpb24gdG8gaGVscCB1bmRlcnN0YW5kIGFuZCBhbmFseXplIHRoZSBkYXRhc2V0IGJldHRlciBhbmQgaW1wcm92ZSBhbGdvcml0aG0gcGVyZm9ybWFuY2UuDQoNCldlIHdpbGwgdXNlIHRoZSA3IHJlZ2lvbnMgYXMgZGVmaW5lZCBpbiB0aGUgV29ybGQgQmFuayBEZXZlbG9wbWVudCBJbmRpY2F0b3JzLiBUaGVzZSByZWdpb25zIGFyZToNCg0KMS4gIEVhc3QgQXNpYSBhbmQgUGFjaWZpYzogVGhpcyByZWdpb24gaW5jbHVkZXMgY291bnRyaWVzIGxpa2UgQ2hpbmEsIEF1c3RyYWxpYSwgSW5kb25lc2lhLCBUaGFpbGFuZCwgZXRjLg0KDQoyLiAgRXVyb3BlIGFuZCBDZW50cmFsIEFzaWE6IFRoaXMgcmVnaW9uIGluY2x1ZGVzIGNvdW50cmllcyBsaWtlIEdlcm1hbnksIFVLLCBSdXNzaWEsIFR1cmtleSwgZXRjLg0KDQozLiAgTGF0aW4gQW1lcmljYSAmIENhcmliYmVhbjogVGhpcyByZWdpb24gaW5jbHVkZXMgY291bnRyaWVzIGxpa2UgQnJhemlsLCBNZXhpY28sIEFyZ2VudGluYSwgQ3ViYSwgZXRjLg0KDQo0LiAgTWlkZGxlIEVhc3QgYW5kIE5vcnRoIEFmcmljYTogVGhpcyByZWdpb24gaW5jbHVkZXMgY291bnRyaWVzIGxpa2UgU2F1ZGkgQXJhYmlhLCBFZ3lwdCwgSXJhbiwgSXJhcSwgZXRjLg0KDQo1LiAgTm9ydGggQW1lcmljYTogVGhpcyBpcyBwcmVkb21pbmFudGx5IFVuaXRlZCBTdGF0ZXMgYW5kIENhbmFkYS4NCg0KNi4gIFNvdXRoIEFzaWE6IFRoaXMgcmVnaW9uIGluY2x1ZGVzIGNvdW50cmllcyBsaWtlIEluZGlhLCBQYWtpc3RhbiwgQmFuZ2xhZGVzaCwgU3JpIExhbmthLCBldGMuDQoNCjcuICBTdWItU2FoYXJhbiBBZnJpY2E6IFRoaXMgcmVnaW9uIGluY2x1ZGVzIGNvdW50cmllcyBsaWtlIE5pZ2VyaWEsIFNvdXRoIEFmcmljYSwgRXRoaW9waWEsIEtlbnlhLCBldGMuDQoNCk5vdGU6IFVNKFRoZSBVbml0ZWQgU3RhdGVzIE1pbm9yIE91dGx5aW5nIElzbGFuZHMpIGFuZCBBUShBbnRhcmN0aWNhKSBkb24ndCBiZWxvbmcgdG8gYW55IG9mIHRoZXNlIHJlZ2lvbnMsIHRodXMsIHRoZXkgd2lsbCBiZSB1c2VkIGFzIHRoZXkgYXJlLg0KDQpgYGB7cn0NCg0KDQp1bT13aGljaChkYXRhc2V0JGNvbXBhbnlfbG9jYXRpb249PSJVTSIpDQphcT13aGljaChkYXRhc2V0JGNvbXBhbnlfbG9jYXRpb249PSJBUSIpDQoNCg0KZGF0YXNldCRjb21wYW55X2xvY2F0aW9uIDwtIGNvdW50cnljb2RlKGRhdGFzZXQkY29tcGFueV9sb2NhdGlvbiwgImlzbzJjIiwgInJlZ2lvbiIpDQpkYXRhc2V0JGVtcGxveWVlX3Jlc2lkZW5jZSA8LSBjb3VudHJ5Y29kZShkYXRhc2V0JGVtcGxveWVlX3Jlc2lkZW5jZSwgImlzbzJjIiwgInJlZ2lvbiIpDQoNCmRhdGFzZXRbdW0sImNvbXBhbnlfbG9jYXRpb24iXT0iVU0iDQpkYXRhc2V0W2FxLCJjb21wYW55X2xvY2F0aW9uIl09IkFRIg0KDQpgYGANCg0KQ29uY2VwdCBoaWVyYXJjaHkgZ2VuZXJhdGlvbiBjYW4gYmUgZG9uZSBmb3IgImpvYl90aXRsZSIgYXMgd2VsbCB0byBpbXByb3ZlIGludGVycHJldGF0aW9uIGFuZCBzY2FsYWJpbGl0eS4gQWxzbywgbW9zdCBqb2IgdGl0bGVzIGFyZSBlc3NlbnRpYWxseSB0aGUgc2FtZSBqb2IgYnV0IHdpdGggZGlmZmVyZW50IG5hbWVzLCBzbyB3ZSBjYW4gY29tYmluZSB0aGVtIGludG8gYSBoaWdoZXItbGV2ZWwgam9icyB0aXRsZXMgc3VjaCBhcyBBcmNoaXRlY3QsIEFuYWx5c3QgYW5kIEVuZ2luZWVyIGV0Yy4NCg0KYGBge3J9DQojIyBDcmVhdGUgdGhlIGNhdGVnb3JpZXMgYmFzZWQgb24gam9iIHJhbmsgDQpkYXRhc2V0JGpvYl90aXRsZSA8LSBpZmVsc2UoZ3JlcGwoIkFuYWx5c3QiLCBkYXRhc2V0JGpvYl90aXRsZSksICJBbmFseXN0IiwNCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgaWZlbHNlKGdyZXBsKCJBcmNoaXRlY3QiLCBkYXRhc2V0JGpvYl90aXRsZSksICJBcmNoaXRlY3QiLA0KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgaWZlbHNlKGdyZXBsKCJFbmdpbmVlciIsIGRhdGFzZXQkam9iX3RpdGxlKSwgIkVuZ2luZWVyIiwNCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBpZmVsc2UoZ3JlcGwoIk1hbmFnZXJ8T2ZmaWNlcnxEaXJlY3RvcnxMZWFkZXIiLCBkYXRhc2V0JGpvYl90aXRsZSksICJMZWFkZXJzaGlwIiwNCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgaWZlbHNlKGdyZXBsKCJDb25zdWx0YW50fFNwZWNpYWxpc3QiLCBkYXRhc2V0JGpvYl90aXRsZSksICJDb25zdWx0YW50L1NwZWNpYWxpc3QiLA0KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgaWZlbHNlKGdyZXBsKCJDeWJlciIsIGRhdGFzZXQkam9iX3RpdGxlKSwgIkN5YmVyIFNlY3VyaXR5IiwNCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAiT3RoZXJzIikpKSkpKQ0KDQpgYGANCg0KIyMgRW5jb2RpbmcgY2F0ZWdvcmljYWwgZGF0YQ0KDQpUbyBkZWFsIHdpdGggY29sdW1ucyB3aXRoIGNoYXJhY3RlciB0eXBlIHdlIGFyZSBnb2luZyB0byBlbmNvZGUgdGhlbSwgYmVjYXVzZSBtb3N0IG1hY2hpbmUgbGVhcm5pbmcgYWxnb3JpdGhtcyBhcmUgZGVzaWduZWQgdG8gd29yayB3aXRoIGZhY3RvcnMgZGF0YSByYXRoZXIgdGhhbiBjaGFyYWN0ZXIgZGF0YSBhbmQgaXQgaW1wcm92ZXMgcGVyZm9ybWFuY2UgYW5kIEludGVycHJldGFiaWxpdHkgb2YgZGF0YSBhcyB3ZWxsLg0KDQpgYGB7cn0NCmRhdGFzZXQkam9iX3RpdGxlICA8LSBmYWN0b3IoZGF0YXNldCRqb2JfdGl0bGUpDQoNCmRhdGFzZXQkZXhwZXJpZW5jZV9sZXZlbCA9IGZhY3RvcihkYXRhc2V0JGV4cGVyaWVuY2VfbGV2ZWwsIGxldmVscz1jKCJFTiIsICJNSSIsICJTRSIsICJFWCIpLCBsYWJlbHM9YygxLDIsMyw0KSkNCg0KZGF0YXNldCRlbXBsb3ltZW50X3R5cGUgIDwtIGZhY3RvcihkYXRhc2V0JGVtcGxveW1lbnRfdHlwZSkNCg0KZGF0YXNldCRlbXBsb3llZV9yZXNpZGVuY2UgIDwtIGZhY3RvcihkYXRhc2V0JGVtcGxveWVlX3Jlc2lkZW5jZSkNCg0KZGF0YXNldCRjb21wYW55X2xvY2F0aW9uICA8LSBmYWN0b3IoZGF0YXNldCRjb21wYW55X2xvY2F0aW9uKQ0KDQpkYXRhc2V0JHNhbGFyeV9jdXJyZW5jeSAgPC0gZmFjdG9yKGRhdGFzZXQkc2FsYXJ5X2N1cnJlbmN5KQ0KDQpkYXRhc2V0JGpvYl90aXRsZSAgPC0gZmFjdG9yKGRhdGFzZXQkam9iX3RpdGxlKQ0KDQoNCmRhdGFzZXQkY29tcGFueV9zaXplID0gZmFjdG9yKGRhdGFzZXQkY29tcGFueV9zaXplLCBsZXZlbHM9YygiUyIsIk0iLCJMIiksIGxhYmVscz1jKDEsMiwzKSkNCg0KDQpkYXRhc2V0JGpvYl90aXRsZSAgPC0gZmFjdG9yKGRhdGFzZXQkam9iX3RpdGxlKQ0KDQpgYGANCg0KIyMjIERpc2NyZXRpemF0aW9uIG9mIHNhbGFyYXlfaW5fdXNkIGF0dHJpYnV0ZQ0KDQpieSBjYWxjdWxhdGluZyBicmVha3MgYmFzZWQgb24gcXVhcnRpbGVzDQoNCmBgYHtyfQ0KYnJlYWtzIDwtIHF1YW50aWxlKGRhdGFzZXQkc2FsYXJ5X2luX3VzZCwgDQogICAgICAgICAgICAgICAgICAgcHJvYnMgPSBjKDAsIC4yNSwgLjUsIC43NSwgLjk1LCAxKSwgDQogICAgICAgICAgICAgICAgICAgbmEucm0gPSBUUlVFKQ0KDQoNCmRhdGFzZXQkc2FsYXJ5X2luX3VzZCA8LSBjdXQoZGF0YXNldCRzYWxhcnlfaW5fdXNkLCANCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGJyZWFrcyA9IGJyZWFrcywgDQogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBpbmNsdWRlLmxvd2VzdCA9IFRSVUUsIA0KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgbGFiZWxzPWMoIlZlcnkgTG93IiwgIkxvdyIsICJNZWRpdW0iLCAiSGlnaCIsICJWZXJ5IEhpZ2giKSkNCg0KDQpgYGANCg0KIyMjIE5vcm1hbGl6YXRpb246DQoNCnRvIGNoYW5nZSB0aGUgc2NhbGUgb2YgbnVtZXJpYyBhdHRyaWJ1dGVzIChyZW1vdGVfcmF0aW8gYW5kIHdvcmtfeWVhcikgdG8gYSBzY2FsZSBvZiBbLTEsMV0gdG8gZ2l2ZSB0aGVtIGVxdWFsIHdlaWdodA0KDQpgYGB7cn0NCmRhdGFzZXQgWywgYygid29ya195ZWFyIiAsICJyZW1vdGVfcmF0aW8iKV0gPSBzY2FsZShkYXRhc2V0IFssIGMoIndvcmtfeWVhciIgLCAicmVtb3RlX3JhdGlvIildKQ0KYGBgDQoNCiMjIEZlYXR1cmUgU2VsZWN0aW9uDQoNCndlIHdpbGwgaW1wbGVtZW50IGZlYXR1cmUgc2VsZWN0aW9uIHRvIHJlbW92ZSByZWR1bmRhbnQgb3IgaXJyZWxldmFudCBhdHRyaWJ1dGVzIGZyb20gdGhlIGRhdGEgc2V0IHRvIGdldCB0aGUgc21hbGxlc3Qgc3Vic2V0IHRoYXQgY2FuIGhlbHAgdXMgZ2V0IHRoZSBtb3N0IGFjY3VyYXRlIHByZWRpY3Rpb25zIGZvciBvdXIgdGFyZ2V0IGNsYXNzKHNhbGFyeV9pbl91c2QpIGFuZCBkZWNyZWFzZSB0aGUgdGltZSB0aGF0IGl0IHRha2VzIHRoZSBjbGFzc2lmaWVyIHRvIHByb2Nlc3MgdGhlIGRhdGEuDQoNCndlIHdpbGwgdXNlIFJGRShSZWN1cnNpdmUgZmVhdHVyZSBlbGltaW5hdGlvbikgd2hpY2ggaXMgYSB3cmFwcGVyIG1ldGhvZCBmb3IgdGhlIGZlYXR1cmUgc2VsZWN0aW9uLiBTaW5jZSB0aGUgUkZFIGZ1bmN0aW9uIGhhdmUgbXVsdGlwbGUgY29udHJvbCBvcHRpb25zIHdlIG5lZWQgdG8gc3BlY2lmeSB0aGUgb3B0aW9ucyB0aGF0IHdlIHdhbnQuIFdlIHdpbGwgY2hvb3NlICJSYW5kb20gRm9yZXN0IiBiZWNhdXNlIGl0IGhhcyBoaWdoIGFjY3VyYWN5LCBjYW4gaGFuZGxlIGNhdGVnb3JpY2FsIGRhdGEuDQoNCmBgYHtyfQ0KY29udHJvbCA8LSByZmVDb250cm9sKGZ1bmN0aW9ucyA9IHJmRnVuY3MsIA0KICAgICAgICAgICAgICAgICAgICAgIG1ldGhvZCA9ICJyZXBlYXRlZGN2IiwNCiAgICAgICAgICAgICAgICAgICAgICByZXBlYXRzID0gNSwgDQogICAgICAgICAgICAgICAgICAgICAgbnVtYmVyID0gMTApDQpgYGANCg0KRmlyc3Qgd2Ugc2F2ZSB0aGUgZmVhdHVyZXMgdG8gYmUgdXNlZCBpbiB0aGUgZmVhdHVyZSBzZWxlY3Rpb24oZXZlcnkgYXR0cmlidXRlcyBleGNlcHQgdGhlIGNsYXNzIGxhYmVsICJzYWxhcnlfaW5fdXNkIikgaW4gdmFyaWFibGUgeCwgYW5kIHRoZSBjbGFzcyBsYWJlbCBpbiB2YXJpYWJsZSB5LiBUaGVuIHNwbGl0IHRoZSBkYXRhIHRvIDgwJSB0cmFpbmluZyBhbmQgMjAlIHRlc3QuDQoNCmBgYHtyfQ0KeCA8LSBkYXRhc2V0ICU+JQ0KICBzZWxlY3QoLXNhbGFyeV9pbl91c2QpICU+JQ0KICBhcy5kYXRhLmZyYW1lKCkNCg0KIyBUYXJnZXQgdmFyaWFibGUNCnkgPC0gZGF0YXNldCRzYWxhcnlfaW5fdXNkDQoNCiMgVHJhaW5pbmc6IDgwJTsgVGVzdDogMjAlDQpzZXQuc2VlZCgyMDIxKQ0KaW5UcmFpbiA8LSBjcmVhdGVEYXRhUGFydGl0aW9uKHksIHAgPSAuODAsIGxpc3QgPSBGQUxTRSlbLDFdDQoNCnhfdHJhaW4gPC0geFsgaW5UcmFpbiwgXQ0KeF90ZXN0ICA8LSB4Wy1pblRyYWluLCBdDQoNCnlfdHJhaW4gPC0geVsgaW5UcmFpbl0NCnlfdGVzdCAgPC0geVstaW5UcmFpbl0NCg0KYGBgDQoNCmFmdGVyIHNwbGl0dGluZyB0aGUgZGF0YSwgbm93IHdlIGNhbiBwZXJmb3JtIHRoZSBzZWxlY3Rpb24gdXNpbmcgcmZlDQoNCmBgYHtyfQ0Kc2V0LnNlZWQoMSkNCnJlc3VsdF9yZmUxIDwtIHJmZSh4ID0geF90cmFpbiwgDQogICAgICAgICAgICAgICAgICAgeSA9IHlfdHJhaW4sIA0KICAgICAgICAgICAgICAgICAgIHNpemVzID0gYygxOjkpLA0KICAgICAgICAgICAgICAgICAgIHJmZUNvbnRyb2wgPSBjb250cm9sKQ0KDQpyZXN1bHRfcmZlMQ0KDQpwcmVkaWN0b3JzKHJlc3VsdF9yZmUxKQ0KDQpgYGANCg0KVGhlIHJlc3VsdHMgc2hvdyB0aGF0IGFsbCB0aGUgcmVtYWluaW5nIGF0dHJpYnV0ZXMsIGV4Y2VwdCBmb3IgImVtcGxveW1lbnRfdHlwZSIsIGFyZSBzZWxlY3RlZC4gVGhpcyBpcyBsb2dpY2FsLCBhcyA5OCUgb2YgdGhlIHJvd3MgaGF2ZSB0aGUgdmFsdWUgIkZUIiwgYXMgc2hvd24gaW4gdGhlIHRhYmxlIGJlbG93LiBEdWUgdG8gdGhlIGxvdyB2YXJpYW5jZSwgd2UgZGVjaWRlZCB0byByZW1vdmUgdGhpcyBhdHRyaWJ1dGUuDQoNCmBgYHtyfQ0KdGFibGUoZGF0YXNldCRlbXBsb3ltZW50X3R5cGUpDQpgYGANCg0KYGBge3J9DQpkYXRhc2V0PC1kYXRhc2V0Wywtd2hpY2gobmFtZXMoZGF0YXNldCk9PSJlbXBsb3ltZW50X3R5cGUiKV0NCmBgYA0KDQojIHBoYXNlIDMNCg0KRHVyaW5nIHRoaXMgcGhhc2UsIG91ciBmb2N1cyB3aWxsIGJlIG9uIGNsdXN0ZXJpbmcgYW5kIGNsYXNzaWZpY2F0aW9uIHRlY2huaXF1ZXMgdG8gYW5hbHl6ZSB0aGUgZGF0YS4gVGhlIHByaW1hcnkgb2JqZWN0aXZlcyBhcmUgdG8gaWRlbnRpZnkgZGlzdGluY3QgZ3JvdXBzIHdpdGhpbiB0aGUgZGF0YXNldCB0aHJvdWdoIGNsdXN0ZXJpbmcsIGNsYXNzaWZ5IGRhdGEgb2JqZWN0cyBpbnRvIG1lYW5pbmdmdWwgY2F0ZWdvcmllcywgYW5kIGFwcGx5IGRpZmZlcmVudCBldmFsdWF0aW9uIG1ldGhvZHMgdG8gYXNzZXNzIHRoZSBhY2N1cmFjeSBhbmQgcHJlY2lzaW9uIG9mIGJvdGggY2xhc3NpZmljYXRpb24gYW5kIGNsdXN0ZXJpbmcgcmVzdWx0cy4gV2UgYWltIHRvIGdhaW4gZGVlcGVyIGluc2lnaHRzIGludG8gdGhlIGRhdGEgYW5kIGRpc2NvdmVyIHBhdHRlcm5zLg0KDQojIyBSZXRyZWl2ZSBvdXIgcHJlcHJvY2Vzc2VkIGRhdGFzZXQNCg0KYGBge3J9DQoNCiMgUmVhZCB0aGUgQ1NWIGZpbGUgZnJvbSBnaXRodWINCmRhdGFzZXQyPSByZWFkLmNzdih1cmwoImh0dHBzOi8vcmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbS9TYXJhaEFsaGluZGkvRE1fcHJvamVjdC9tYWluL0RhdGElMjBTZXQvcHJlcHJvY2Vzc2VkRGF0YXNldC5jc3YiKSwgaGVhZGVyPVRSVUUpDQoNCiMgSWRlbnRpZnkgdGhlIGNoYXJhY3RlciB2YXJpYWJsZXMgaW4gdGhlIGRhdGFzZXQyDQpjaGFyX3ZhcnMgPC0gc2FwcGx5KGRhdGFzZXQyLCBpcy5jaGFyYWN0ZXIpDQoNCiMgQ29udmVydCB0aGUgaWRlbnRpZmllZCBjaGFyYWN0ZXIgdmFyaWFibGVzIGluIGRhdGFzZXQyIHRvIGZhY3RvcnMNCmRhdGFzZXQyW2NoYXJfdmFyc10gPC0gbGFwcGx5KGRhdGFzZXQyW2NoYXJfdmFyc10sIGFzLmZhY3RvcikNCg0KYGBgDQoNCiMjIGJhbGFuY2luZyBkYXRhDQoNClRvIHJlc29sdmUgdGhlIHByb2JsZW0gb2YgY2xhc3MgaW1iYWxhbmNlIGluIHRoZSBkYXRhc2V0LCB3ZSB3aWxsIHVzZSBTTU9URSgpIG1ldGhvZCB0aGF0IG92ZXJzYW1wbGUgdGhlIG1pbm9yaXR5IGNsYXNzIGJ5IGNyZWF0aW5nIHN5bnRoZXRpYyBzYW1wbGVzIHVzaW5nIHRoZSBleGlzdGluZyBtaW5vcml0eSBjbGFzcyBzYW1wbGVzDQoNCmBgYHtyfQ0Kc2V0LnNlZWQoMTApDQpiYWxhbmNlZF9kYXRhc2V0IDwtIFNNT1RFKHNhbGFyeV9pbl91c2QgfiAuLCBkYXRhc2V0MiwgcGVyYy5vdmVyID0gMzAwLCBwZXJjLnVuZGVyPTUwMCwgayA9IDEwKQ0KYGBgDQoNCiMjIENsYXNzaWZpY2F0aW9uDQoNClRoZSBnb2FsIG9mIGFsbCBwcmVjZWRpbmcgc3RlcHMgaXMgdG8gcHJvcGVybHkgcHJlcGFyZSB0aGUgZGF0YXNldCBmb3IgdGhlIGNsYXNzaWZpY2F0aW9uIGFuZCBsYXRlciBjbHVzdGVyaW5nIHBoYXNlLCB3aGljaCBjb25zdGl0dXRlcyBvbmUgb2Ygb3VyIHByaW1hcnkgbWluaW5nIG9iamVjdGl2ZXMuIEluIHRoaXMgc2VjdGlvbiwgd2Ugd2lsbCBlbXBsb3kgdmFyaW91cyBhdHRyaWJ1dGUgc2VsZWN0aW9uIG1ldGhvZHMgc3VjaCBhcyB0aGUgR2luaSBpbmRleCwgR2FpbiByYXRpbywgYW5kIGluZm9ybWF0aW9uIGdhaW4gdG8gY29uc3RydWN0IGEgZGVjaXNpb24gdHJlZSBtb2RlbC4gV2Ugd2lsbCB0aG9yb3VnaGx5IGV2YWx1YXRlIGl0cyBwZXJmb3JtYW5jZSwgYW5kIGlmIGl0IHByb3ZlcyBlZmZlY3RpdmUsIGl0IGNhbiBzdWJzZXF1ZW50bHkgYmUgdXRpbGl6ZWQgdG8gY2xhc3NpZnkgbmV3IGluc3RhbmNlcyB3aXRoIHVua25vd24gY2xhc3MgbGFiZWxzLg0KDQpzaW5jZSBvdXIgZGF0YXNldCBpcyBzbWFsbCwgd2UgZGVjaWRlZCB0byB1c2UgSy1mb2xkIENyb3NzLXZhbGlkYXRpb24gYXMgYSBkYXRhc2V0IHBhcnRpb25pbmcgbWV0aG9kLiBmb3IgZWFjaCBhdHRyaWJ1dGUgc2VsZWN0aW9uIG1ldGhvZCB3ZSB3aWxsIHRyeSBkaWZmZXJlbnQgSyBzaXplICgxMCw1LCBhbmQgMykNCg0KaW4gYWxsIHRoaXMgc2VjdGlvbiB3ZSB3aWxsIGJlIHVzaW5nIHRyYWluIGFuZCB0cmFpbkNvbnRyb2wgZnVuY3Rpb25zIG9mIGNhcmV0IHBhY2thZ2UgdG8gcHJvZHVjZSBkZWNpc2lvbiB0cmVlcy4gZm9yIEdpbmkgaW5kZXggdGhlIG1ldGhvZCB3aWxsIGJlICJycGFydCIgYW5kIGZvciBHYWluIHJhdGlvIGl0J3MgImo0OCIgYXMgZm9yIGluZm9ybWF0aW9uIGdhaW4gdGhlIG1ldGhvZCBpcyAiQzUuMCIuDQoNCnRoZSBmb2xsb3dpbmcgZnVuY3Rpb24gd2lsbCBiZSB1c2VkIHRvIGNvbXB1dGUgYXZlcmFnZSBzZW5zaXRpdml0eSBhbmQgU3BlY2lmaWNpdHk6DQoNCmBgYHtyfQ0KDQoNCm1hY3JvID0gZnVuY3Rpb24obWF0cml4KXsNCiAgDQogIHN1bVNlbj0wDQogIA0KICBmb3IgKGkgaW4gMTo1KSB7DQogICBzdW1TZW4gPSBzdW1TZW4gKyBtYXRyaXgkYnlDbGFzc1tpLDFdIA0KICB9DQogIA0KICANCiAgYXZnU2VuID0gc3VtU2VuLzUNCiAgDQogIHN1bVNwZWM9MA0KICANCiAgZm9yIChpIGluIDE6NSkgew0KICAgc3VtU3BlYyA9IHN1bVNwZWMgKyBtYXRyaXgkYnlDbGFzc1tpLDJdIA0KICB9DQogIGF2Z1NwZWMgPSBzdW1TcGVjLzUNCiAgDQogIA0KICANCiAgDQogIHN1bVByZWM9MA0KICANCiAgZm9yIChpIGluIDE6NSkgew0KICAgc3VtUHJlYyA9IHN1bVByZWMgKyBtYXRyaXgkYnlDbGFzc1tpLDNdIA0KICB9DQogIGF2Z1ByZWMgPSBzdW1QcmVjLzUNCiAgDQogIA0KICANCiAgDQogIGF2Z3MgPSBkYXRhLmZyYW1lKFNlbnNpdGl2aXR5PWF2Z1NlbiAsIFNwZWNpZmljaXR5PWF2Z1NwZWMsIFByZWNpc2lvbj1hdmdQcmVjICxBY2N1cmFjeT0gdW5uYW1lKCBtYXRyaXgkb3ZlcmFsbFsxXSkgKQ0KICBwcmludChhdmdzKQ0KICANCiAgDQp9DQoNCg0KYGBgDQoNCiMjIyBHaW5pIGluZGV4DQoNCkdpbmkgaW5kZXggbWVhc3VyZXMgdGhlIGltcHVyaXR5IG9mIHRoZSBkYXRhc2V0LiBUaGUgcGFydGl0aW9uaW5nIHRoYXQgeWllbGRzIHRoZSBtb3N0IHN1YnN0YW50aWFsIHJlZHVjdGlvbiBpbiBpbXB1cml0eSBpcyBzZWxlY3RlZCBhcyB0aGUgc3BsaXR0aW5nIGF0dHJpYnV0ZS4gVG8gYXBwbHkgdGhlIEdpbmkgaW5kZXgsIHdlIHdpbGwgZW1wbG95IHRoZSAicnBhcnQiIG1ldGhvZCwgd2hpY2ggdXRpbGl6ZXMgdGhlIEdpbmkgaW5kZXggYXMgdGhlIGNyaXRlcmlhIGZvciBzcGxpdHRpbmcuDQoNCg0KIyMjIyAxMCBGb2xkcw0KDQpUaGUgdHJlZSBvZiB0aGUgZ2luaSBpbmRleCB1c2luZyAxMCBmb2xkcw0KDQpgYGB7cn0NCg0Kc2V0LnNlZWQoMTApDQpjdHJsIDwtIHRyYWluQ29udHJvbChtZXRob2QgPSAiY3YiLCBudW1iZXIgPSAxMCwgcmV0dXJuUmVzYW1wPSJhbGwiLCBzYXZlUHJlZGljdGlvbnM9ImZpbmFsIikNCg0KdHVuZUdyaWQgPC0gZXhwYW5kLmdyaWQoY3AgPSBjKDAuMDAxLCAwLjAwNSwgMC4wMSkpDQoNCmdpbmlJbmRleDEwIDwtIHRyYWluKA0KICBzYWxhcnlfaW5fdXNkIH4gLiwNCiAgZGF0YSA9IGJhbGFuY2VkX2RhdGFzZXQsDQogIG1ldGhvZCA9ICJycGFydCIsDQogIHRyQ29udHJvbCA9IGN0cmwsdHVuZUdyaWQ9dHVuZUdyaWQsDQogIGNvbnRyb2wgPSBsaXN0KA0KICAgIG1pbnNwbGl0ID0gMTAsDQogICAgbWluYnVja2V0ID0gNSwNCiAgICB4dmFsID0gMTAsDQogICAgY3AgPSAwLjAwMDENCiAgKQ0KDQopDQoNCg0KcHJwKGdpbmlJbmRleDEwJGZpbmFsTW9kZWwsIGJveC5wYWxldHRlID0gIlJlZHMiLCB0d2VhayA9IDEuMiwgdmFybGVuID0gMjApDQoNCmBgYA0KDQoNCnRoZSAiZXhwZXJpbmNlIGxldmVsIiBhdHRyaWJ1dGUgd2FzIHNlbGVjdGVkIGFzIHRoZSBmaXJzdCBzcGxpdHRpbmcgYXR0cmlidXRlIG1lYW5pbmcgdGhhdCBpdCBoYXMgdGhlIGxhcmdlc3QgaW1wdXJpdHkgcmVkdWN0aW9uLg0KDQoNCg0KIyMjIyMgQ29uZnVzaW9uIG1hdHJpeCBvZiAxMCBmb2xkcyB1c2luZyBHaW5pIEluZGV4DQoNClRoZSBmb2xsb3dpbmcgY29uZnVzaW9uIE1hdHJpeCB3aWxsIHNob3cgdGhlIHBlcmZvcm1hbmNlIG9mIHRoZSBjbGFzc2lmaWVyIHVzaW5nIHRoZSBwcmVkaWN0ZWQgY2xhc3MgbGFiZWxzIGFuZCB0aGUgYWN0dWFsIGNsYXNzIGxhYmVscyBvZiBvdXIgZGF0YXNldA0KDQpgYGB7cn0NCg0KZ2luaUluZGV4MTBjbSA9IGNhcmV0Ojpjb25mdXNpb25NYXRyaXgoZ2luaUluZGV4MTAkcHJlZCRvYnMsZ2luaUluZGV4MTAkcHJlZCRwcmVkKQ0KDQpnaW5pSW5kZXgxMGNtDQoNCmBgYA0KDQp0aGUgbWV0cmljcyBzaG93biBmb3IgZWFjaCBjbGFzcyBpbmRpY2F0ZSB0aGUgdmFsdWUgb2YgdGhhdCBtZXRyaWMgd2hlbiB0cmVhdGluZyB0aGlzIGNsYXNzIGFzIHRoZSBwb3NpdGl2ZSBjbGFzcyBhbmQgdGhlIG90aGVyIGNsYXNzZXMgYXMgdGhlIG5lZ2F0aXZlIGNsYXNzLiBoZXJlIHRoZSBjbGFzc2lmaWVyIHNob3dlZCBiZXN0IHBlcmZvcm1hbmNlIHdoZW4gdXNpbmcgdGhlICJ2ZXJ5IGhpZ2giIGNsYXNzIGFzIHRoZSBwb3NpdGl2ZSBjbGFzcyBidXQgdGhpcyB2YWx1ZSBpbiBpdHMgb3duIGRvZXNuJ3QgaG9sZCBtdWNoIHZhbHVlIHNpbmNlIGFsbCBjbGFzc2VzIHNob3VsZCBiZSB0YWtlbiBpbnRvIGNvbnNpZGVyYXRpb24uDQoNCg0KDQoNCg0KDQoNCiMjIyMgNSBGb2xkcw0KVGhlIHRyZWUgb2YgdGhlIGdpbmkgaW5kZXggdXNpbmcgNSBmb2xkcw0KDQpgYGB7cn0NCnNldC5zZWVkKDEwKQ0KY3RybCA8LSB0cmFpbkNvbnRyb2wobWV0aG9kID0gImN2IiwgbnVtYmVyID0gNSwgcmV0dXJuUmVzYW1wPSJhbGwiLCBzYXZlUHJlZGljdGlvbnM9ImZpbmFsIikNCg0KDQp0dW5lR3JpZCA8LSBleHBhbmQuZ3JpZChjcCA9IGMoMC4wMDEsIDAuMDA1LCAwLjAxKSkNCg0KZ2luaUluZGV4NSA8LSB0cmFpbihzYWxhcnlfaW5fdXNkIH4gLiwgZGF0YSA9IGJhbGFuY2VkX2RhdGFzZXQsIG1ldGhvZCA9ICJycGFydCIsIHRyQ29udHJvbCA9IGN0cmwsdHVuZUdyaWQ9dHVuZUdyaWQsDQogIGNvbnRyb2wgPSBsaXN0KA0KICAgIG1pbnNwbGl0ID0gMTAsDQogICAgbWluYnVja2V0ID0gNSwNCiAgICB4dmFsID0gMTAsDQogICAgY3AgPSAwLjAwMDENCiAgKSkNCg0KcHJwKGdpbmlJbmRleDUkZmluYWxNb2RlbCwgYm94LnBhbGV0dGUgPSAiUmVkcyIsIHR3ZWFrID0gMS41LCB2YXJsZW4gPSAxMCwgY2V4ID0gMC4xNSkNCg0KDQpgYGANCg0KdGhpcyB0cmVlIGhhcyB0aGUgc2FtZSBzdHJ1Y3R1cmUgYXMgdGhlIHByZXZpb3VzIHRyZWUgdGhhdCB1c2VkIDEwIGZvbGRzLiBzbyBpbiB0aGlzIHRyZWUgYXMgd2VsbCAiZXhwZXJpZW5jZSBsZXZlbCIgd2FzIGNob29zZSBhcyB0aGUgZmlyc3Qgc3BsaXR0aW5nIGF0dHJpYnV0ZQ0KDQoNCg0KDQojIyMjIyBDb25mdXNpb24gbWF0cml4IG9mIDUgZm9sZHMgdXNpbmcgR2luaSBJbmRleA0KDQpUaGUgZm9sbG93aW5nIGNvbmZ1c2lvbiBNYXRyaXggd2lsbCBzaG93IHRoZSBwZXJmb3JtYW5jZSBvZiB0aGUgY2xhc3NpZmllciB1c2luZyB0aGUgcHJlZGljdGVkIGNsYXNzIGxhYmVscyBhbmQgdGhlIGFjdHVhbCBjbGFzcyBsYWJlbHMgb2Ygb3VyIGRhdGFzZXQNCg0KYGBge3J9DQpnaW5pSW5kZXg1Y20gPSBjYXJldDo6Y29uZnVzaW9uTWF0cml4KGdpbmlJbmRleDUkcHJlZCRvYnMsZ2luaUluZGV4NSRwcmVkJHByZWQpDQoNCmdpbmlJbmRleDVjbQ0KDQpgYGANCg0KdGhlIHJlc3VsdHMgYXJlIHZlcnkgY2xvc2UgdG8gdGhlIDEwIGZvbGRzIHRyZWUsIHNvIGhlcmUgYXMgd2VsbCB0aGUgY2xhc3NpZmllciBzaG93cyBiZXR0ZXIgcGVyZm9ybWFuY2Ugd2hlbiBkZWFsaW5nIHdpdGggdGhlICJ2ZXJ5IGhpZ2giDQoNCg0KDQoNCg0KDQoNCiMjIyMgMyBGb2xkcw0KDQpUaGUgdHJlZSBvZiB0aGUgZ2luaSBpbmRleCB1c2luZyAzIGZvbGRzDQoNCmBgYHtyfQ0Kc2V0LnNlZWQoMTApDQpjdHJsIDwtIHRyYWluQ29udHJvbChtZXRob2QgPSAiY3YiLCBudW1iZXIgPSAzLCByZXR1cm5SZXNhbXA9ImFsbCIsIHNhdmVQcmVkaWN0aW9ucz0iZmluYWwiKQ0KDQoNCnR1bmVHcmlkIDwtIGV4cGFuZC5ncmlkKGNwID0gYygwLjAwMSwgMC4wMDUsIDAuMDEpKQ0KDQpnaW5pSW5kZXgzIDwtIHRyYWluKHNhbGFyeV9pbl91c2QgfiAuLCBkYXRhID0gYmFsYW5jZWRfZGF0YXNldCwgbWV0aG9kID0gInJwYXJ0IiwgdHJDb250cm9sID0gY3RybCx0dW5lR3JpZD10dW5lR3JpZCwNCiAgY29udHJvbCA9IGxpc3QoDQogICAgbWluc3BsaXQgPSAxMCwNCiAgICBtaW5idWNrZXQgPSA1LA0KICAgIHh2YWwgPSAxMCwNCiAgICBjcCA9IDAuMDAwMQ0KICApKQ0KDQpwcnAoZ2luaUluZGV4MyRmaW5hbE1vZGVsLCBib3gucGFsZXR0ZSA9ICJSZWRzIiwgdHdlYWsgPSAxLjUsIHZhcmxlbiA9IDEwLCBjZXggPSAwLjE1KQ0KDQoNCmBgYA0KDQpUaGUgdHJlZSBzaG93cyBzaW1pbGFyIHN0cnVjdHVyZSBhcyB0aGUgdHdvIHByZXZpb3VzIHR3byB0cmVlcywgd2hldGhlciBpdCdzIGluIGl0cyBjaG9vc2Ugb2YgdGhlIHNwbGl0dGluZyBhdHRyaWJ1dGVzIG9yIHRoZSBsZWF2ZXMuDQoNCg0KDQoNCg0KIyMjIyMgQ29uZnVzaW9uIG1hdHJpeCBvZiAzIGZvbGRzIHVzaW5nIEdpbmkgSW5kZXgNCg0KVGhlIGZvbGxvd2luZyBjb25mdXNpb24gTWF0cml4IHdpbGwgc2hvdyB0aGUgcGVyZm9ybWFuY2Ugb2YgdGhlIGNsYXNzaWZpZXIgdXNpbmcgdGhlIHByZWRpY3RlZCBjbGFzcyBsYWJlbHMgYW5kIHRoZSBhY3R1YWwgY2xhc3MgbGFiZWxzIG9mIG91ciBkYXRhc2V0DQoNCmBgYHtyfQ0KDQpnaW5pSW5kZXgzY20gPSBjYXJldDo6Y29uZnVzaW9uTWF0cml4KGdpbmlJbmRleDMkcHJlZCRvYnMsZ2luaUluZGV4MyRwcmVkJHByZWQpDQoNCmdpbmlJbmRleDNjbQ0KDQpgYGANCg0KaGVyZSBhcyB3ZWxsIHRoZSAidmVyeSBoaWdoIiBjbGFzcyBoYXMgdGhlIGJlc3Qgb3ZlcmFsbCBwZXJmb3JtYW5jZQ0KDQoNCg0KDQoNCg0KIyMjIyMgQW5hbHlzaXMgb2YgdGhlIGdpbmkgaW5kZXggY2xhc3NpZmljYXRpb24NCkFsbCB0aHJlZSB0cmVlcyBzZWVtIHRvIGJlIGFsaWtlIGluIHRoZWlyIA0KYXJyYW5nZW1lbnQgYW5kIGZvcm0uDQoNCjEuIFJvb3QgTm9kZSAtIEV4cGVyaWVuY2UgTGV2ZWw6IFRoZSBpbml0aWFsIGF0dHJpYnV0ZSB1c2VkIGZvciBzcGxpdHRpbmcgdGhlIGRhdGFzZXQgYXQgdGhlIHJvb3Qgbm9kZSBpcyB0aGUgImV4cGVyaWVuY2UgbGV2ZWwuIiANClRoaXMgZGl2aWRlcyB0aGUgdHJlZSBpbnRvIHR3byBtYWluIGJyYW5jaGVzIG9yIHN1YnRyZWVzOg0KICAgIC0gUmlnaHQgU3VidHJlZTogVGhpcyBjb21wcmlzZXMgaW5zdGFuY2VzIHdpdGggU2VuaW9yIChTRSkgYW5kIEV4ZWN1dGl2ZSAoRVgpIGV4cGVyaWVuY2UgbGV2ZWxzLg0KICAgIC0gTGVmdCBTdWJ0cmVlOiBUaGlzIGluY2x1ZGVzIGluZGl2aWR1YWxzIHdpdGggRW50cnkgKEVOKSBhbmQgTWlkIChNSSkgZXhwZXJpZW5jZSBsZXZlbHMuDQoNCjIuIFJpZ2h0IFN1YnRyZWUgLSB3b3JrIHllYXI6IFRoZSBuZXh0IGF0dHJpYnV0ZSB1c2VkIHRvIGZ1cnRoZXIgY2xhc3NpZnkgdGhlIHJpZ2h0IHN1YnRyZWUgaXMgIndvcmsgeWVhci4iIA0KVGhlIGRlY2lzaW9uIGNyaXRlcmlvbiBpczoNCiAgICAtIElmIHdvcmsgeWVhciBpcyA8LTEuODogVGhlbiBpdCBpcyBoaWdoLg0KICAgIC0gSWYgd29yayB5ZWFyIGlzIE5PVCA8LTEuODogVGhlIG5leHQgYXR0cmlidXRlIGV4YW1pbmVkIGlzICJleHBlcmllbmNlIGxldmVsLiINCg0KMy4gTGVmdCBTdWJ0cmVlIC0gRXhwZXJpZW5jZSBMZXZlbDogT24gdGhlIGxlZnQgc2lkZSBvZiB0aGUgdHJlZSwgdGhlIGF0dHJpYnV0ZSAiZXhwZXJpZW5jZSBsZXZlbC4iIGlzIHVzZWQgdG8gZnVydGhlciBiaWZ1cmNhdGUgdGhlIGluc3RhbmNlczoNCiAgICAtIElmIGV4cGVyaWVuY2UgbGV2ZWwgaXMgPj0yOiBUaGUgbmV4dCBhdHRyaWJ1dGUgZXhhbWluZWQgaXMgImV4cGVyaWVuY2UgbGV2ZWwuIg0KICAgIC0gSWYgZXhwZXJpZW5jZSBsZXZlbCBpcyBOT1QgPj0yOiBUaGUgbmV4dCBhdHRyaWJ1dGUgYWxzbyB3aWxsIGV4YW1pbmVkIGlzICJleHBlcmllbmNlIGxldmVsLiINCg0KIyMjIyMgU2Vuc2l0aXZpdHksIEFjY3VyYWN5LCBTcGVjaWZpdHkgYW5kIHByZWNpc2lvbiBvZiBhbGwgMyw1IGFuZCAxMCBmb2xkcyB1c2luZyBHaW5pIEluZGV4DQoNCmBgYHtyfQ0KcmJpbmQoIjEwIEZvbGRzIj1tYWNybyhnaW5pSW5kZXgxMGNtKSwgIjUgRm9sZHMiPW1hY3JvKGdpbmlJbmRleDVjbSksICIzIEZvbGRzIj1tYWNybyhnaW5pSW5kZXgzY20pICApIA0KYGBgDQoNClRoZSBoaWdoZXIgdmFsdWVzIGZvciBzZW5zaXRpdml0eSwgc3BlY2lmaWNpdHksIHByZWNpc2lvbiwgYW5kIGFjY3VyYWN5IGluIHRoZSAxMC1mb2xkIGNhc2UgaW5kaWNhdGUgYmV0dGVyIG92ZXJhbGwgcGVyZm9ybWFuY2UgYWNjb3JkaW5nIHRvIHRoZXNlIG1ldHJpY3MuDQpzbyxHaW5pIEluZGV4IG1vZGVsIHBlcmZvcm1zIGJldHRlciB3aXRoIGEgMTAtZm9sZCBjcm9zcy12YWxpZGF0aW9uIGNvbXBhcmVkIHRvIDUgYW5kIDMgZm9sZHMuDQoNCg0KDQoNCg0KDQojIyMgR2FpbiByYXRpbw0KDQpUaGUgZ2FpbiByYXRpbywgYSBub3JtYWxpemVkIG1lYXN1cmUgb2YgaW5mb3JtYXRpb24gZ2FpbiwgaXMgY2FsY3VsYXRlZCBieSBkaXZpZGluZyBpbmZvcm1hdGlvbiBnYWluIGJ5IHRoZSBzcGxpdCBpbmZvcm1hdGlvbi4gVGhlIGF0dHJpYnV0ZSB0aGF0IHlpZWxkcyB0aGUgaGlnaGVzdCBnYWluIHJhdGlvIGlzIGNob3NlbiBhcyB0aGUgc3BsaXR0aW5nIGF0dHJpYnV0ZS4gVGhlIEM0LjUgYWxnb3JpdGhtIGVtcGxveXMgdGhlIGdhaW4gcmF0aW8uDQoNClRoZSBKNDggaXMgdGhlIEphdmEtYmFzZWQgb3Blbi1zb3VyY2UgaW1wbGVtZW50YXRpb24gb2YgdGhlIEM0LjUgYWxnb3JpdGhtLCBhbmQgaXQgaXMgaW5jbHVkZWQgaW4gdGhlIFdla2EgcGFja2FnZS4gVGhpcyBpbXBsZW1lbnRhdGlvbiBhbGxvd3MgdXNlcnMgdG8gY29udmVuaWVudGx5IGFwcGx5IHRoZSBDNC41IGRlY2lzaW9uIHRyZWUuDQoNCiMjIyMgMTAgRm9sZHMNCg0KVGhlIHRyZWUgb2YgdGhlIGdhaW4gcmF0aW8gdXNpbmcgMTAgZm9sZHMNCg0KYGBge3IgLCBmaWcuaGVpZ2h0PTcwLCBmaWcud2lkdGg9OTB9DQpzZXQuc2VlZCgxMCkNCmN0cmwgPC0gdHJhaW5Db250cm9sKG1ldGhvZCA9ICJjdiIsIG51bWJlciA9IDEwLCByZXR1cm5SZXNhbXA9ImFsbCIsIHNhdmVQcmVkaWN0aW9ucz0iZmluYWwiKQ0KZ2FpblJhdGlvMTAgPC0gdHJhaW4oc2FsYXJ5X2luX3VzZCB+IC4sIGRhdGEgPSBiYWxhbmNlZF9kYXRhc2V0LCBtZXRob2QgPSAiSjQ4Iix0ckNvbnRyb2wgPSBjdHJsKQ0KcGxvdChnYWluUmF0aW8xMCRmaW5hbE1vZGVsKQ0KYGBgDQoNCnRoZSBmaXJzdCBzcGxpdHRpbmcgYXR0cmlidXRlIHRoYXQgd2FzIGNob29zZW4gaXMgdGhlICJFeHBlaXJlbmNlIGxldmVsIiBhdHRyaWJ1dGUgbWVhbmluZyB0aGF0IGl0IHByb2JhYmx5IGhhcyBhIGhpZ2ggaW5mb3JtYXRpb24gZ2FpbiBhbmQgbG93IHNwbGl0SW5mbyhFbnRyb3B5IG9mIGRpc3RyaWJ1dGlvbiBvZiB0dXBsZXMgaW50byBwYXJ0aXRpb24pDQoNCg0KDQoNCiMjIyMjIENvbmZ1c2lvbiBtYXRyaXggb2YgMTAgZm9sZHMgdXNpbmcgR2FpbiByYXRpbw0KDQpUaGUgZm9sbG93aW5nIGNvbmZ1c2lvbiBNYXRyaXggd2lsbCBzaG93IHRoZSBwZXJmb3JtYW5jZSBvZiB0aGUgY2xhc3NpZmllciB1c2luZyB0aGUgcHJlZGljdGVkIGNsYXNzIGxhYmVscyBhbmQgdGhlIGFjdHVhbCBjbGFzcyBsYWJlbHMgb2Ygb3VyIGRhdGFzZXQNCg0KDQpgYGB7cn0NCmdhaW5SYXRpbzEwY20gPSBjYXJldDo6Y29uZnVzaW9uTWF0cml4KGdhaW5SYXRpbzEwJHByZWQkb2JzLCBnYWluUmF0aW8xMCRwcmVkJHByZWQpDQoNCmdhaW5SYXRpbzEwY20NCg0KDQpgYGANCg0KaGVyZSB0aGUgY2xhc3NpZmllciBzaG93cyBiZXR0ZXIgcGVyZm9ybWFuY2Ugd2hlbiB0cmVhdGluZyAidmVyeSBoaWdoIiBhbmQgInZlcnkgbG93IiBhdHRyaWJ1dGVzIGFzIHBvc2l0aXZlIGNsYXNzLiBzaW5jZSB0aGUgInZlcnkgaGlnaCIgY2xhc3MgaXMgYmV0dGVyIGluIFNlbnNpdGl2aXR5IGFuZCAidmVyeSBsb3ciIGlzIGJldHRlciBpbiBTcGVjaWZpY2l0eSBhbmQgcHJlY2lzaW9uIChQb3MgUHJlZCBWYWx1ZSkNCg0KDQoNCiMjIyMgNSBGb2xkcw0KVGhlIHRyZWUgb2YgdGhlIGdhaW4gcmF0aW8gdXNpbmcgNSBmb2xkcw0KDQoNCmBgYHtyICwgZmlnLmhlaWdodD03MCwgZmlnLndpZHRoPTkwfQ0Kc2V0LnNlZWQoMTApDQpjdHJsIDwtIHRyYWluQ29udHJvbChtZXRob2QgPSAiY3YiLCBudW1iZXIgPSA1LCByZXR1cm5SZXNhbXA9ImFsbCIsIHNhdmVQcmVkaWN0aW9ucz0iZmluYWwiKQ0KZ2FpblJhdGlvNSA8LSB0cmFpbihzYWxhcnlfaW5fdXNkIH4gLiwgZGF0YSA9IGJhbGFuY2VkX2RhdGFzZXQsIG1ldGhvZCA9ICJKNDgiLHRyQ29udHJvbCA9IGN0cmwpDQpwbG90KGdhaW5SYXRpbzUkZmluYWxNb2RlbCkNCmBgYA0KDQp0aGUgdHJlZSBpcyBzaW1pbGFyIHRvIHRoZSB0cmVlIHRoYXQgd2FzIHJlc3VsdGVkIGZyb20gMTAgZm9sZHMuIGl0IGhhcyBjaG9vc2UgIkV4cGVyaWVuY2UgbGV2ZWwiIGFzIHRoZSBmaXJzdCBzcGxpdHRpbmcgYXR0cmlidXRlIGFuZCBhbmQgc2VlbSB0byBzaG93IHNpbWlsYXIgYmVoYXZpb3IuDQoNCg0KDQoNCg0KIyMjIyMgQ29uZnVzaW9uIG1hdHJpeCBvZiA1IGZvbGRzIHVzaW5nIEdhaW4gcmF0aW8NCg0KVGhlIGZvbGxvd2luZyBjb25mdXNpb24gTWF0cml4IHdpbGwgc2hvdyB0aGUgcGVyZm9ybWFuY2Ugb2YgdGhlIGNsYXNzaWZpZXIgdXNpbmcgdGhlIHByZWRpY3RlZCBjbGFzcyBsYWJlbHMgYW5kIHRoZSBhY3R1YWwgY2xhc3MgbGFiZWxzIG9mIG91ciBkYXRhc2V0DQoNCmBgYHtyfQ0KDQpnYWluUmF0aW81Y209Y2FyZXQ6OmNvbmZ1c2lvbk1hdHJpeChnYWluUmF0aW81JHByZWQkb2JzLCBnYWluUmF0aW81JHByZWQkcHJlZCkNCg0KZ2FpblJhdGlvNWNtDQoNCmBgYA0KdW5saWtlIHRoZSAxMCBmb2xkcywgaGVyZSB0aGUgY2xhc3NpZmllciBoYXMgdGhlIGJlc3Qgb3ZlcmFsbCBwZXJmb3JtYW5jZSB3aGVuIGNvbnNpZGVyaW5nIHRoZSAidmVyeSBoaWdoIiBhcyB0aGUgcG9zaXRpdmUgY2xhc3MuDQoNCg0KDQoNCg0KDQojIyMjIDMgRm9sZHMNCg0KVGhlIHRyZWUgb2YgdGhlIGdhaW4gcmF0aW8gdXNpbmcgMyBmb2xkcw0KYGBge3IsIGZpZy5oZWlnaHQ9NzAsIGZpZy53aWR0aD05MH0NCnNldC5zZWVkKDEwKQ0KY3RybCA8LSB0cmFpbkNvbnRyb2wobWV0aG9kID0gImN2IiwgbnVtYmVyID0gMywgcmV0dXJuUmVzYW1wPSJhbGwiLCBzYXZlUHJlZGljdGlvbnM9ImZpbmFsIikNCmdhaW5SYXRpbzMgPC0gdHJhaW4oc2FsYXJ5X2luX3VzZCB+IC4sIGRhdGEgPSBiYWxhbmNlZF9kYXRhc2V0LCBtZXRob2QgPSAiSjQ4Iix0ckNvbnRyb2wgPSBjdHJsKQ0KcGxvdChnYWluUmF0aW8zJGZpbmFsTW9kZWwpDQpgYGANCg0KdGhlIHRyZWUgc2hvd3Mgc2ltaWxhciBiZWhhdmlvciBhcyB0aGUgcHJldmlvdXMgMiB0cmVlcyB0aGF0IHJlc3VsdGVkIGZyb20gdXNpbmcgMTAgYW5kIDUgZm9sZHMuDQoNCg0KDQoNCg0KDQojIyMjIyBDb25mdXNpb24gbWF0cml4IG9mIDMgZm9sZHMgdXNpbmcgR2FpbiByYXRpbw0KDQpUaGUgZm9sbG93aW5nIGNvbmZ1c2lvbiBNYXRyaXggd2lsbCBzaG93IHRoZSBwZXJmb3JtYW5jZSBvZiB0aGUgY2xhc3NpZmllciB1c2luZyB0aGUgcHJlZGljdGVkIGNsYXNzIGxhYmVscyBhbmQgdGhlIGFjdHVhbCBjbGFzcyBsYWJlbHMgb2Ygb3VyIGRhdGFzZXQNCg0KYGBge3J9DQpnYWluUmF0aW8zY209Y2FyZXQ6OmNvbmZ1c2lvbk1hdHJpeChnYWluUmF0aW8zJHByZWQkb2JzLCBnYWluUmF0aW8zJHByZWQkcHJlZCkNCg0KZ2FpblJhdGlvM2NtDQoNCmBgYA0KDQpzaW1pbGFyIHRvIHRlaCA1IGZvbGRzLCB0aGUgInZlcnkgaGlnaCIgY2xhc3MgaGFzIHRoZSBiZXN0IG1ldHJpY3MuDQoNCg0KDQoNCg0KIyMjIyMgQW5hbHlzaXMgb2YgdGhlIGdhaW4gcmF0aW8gY2xhc3NpZmljYXRpb24NCg0KVGhlIG9ic2VydmVkIHN0cnVjdHVyZSBvZiB0aGUgdGhyZWUgZGVjaXNpb24gdHJlZXMgc2VlbXMgdG8gYmUgdGhlIHNhbWUgYW5kIGl0IGNhbiBiZSBzdW1tYXJpemVkIGFzIGZvbGxvd3M6DQoNCjEuIFJvb3QgTm9kZSAtIEV4cGVyaWVuY2UgTGV2ZWw6IFRoZSBpbml0aWFsIGF0dHJpYnV0ZSB1c2VkIGZvciBzcGxpdHRpbmcgdGhlIGRhdGFzZXQgYXQgdGhlIHJvb3Qgbm9kZSBpcyB0aGUgImV4cGVyaWVuY2UgbGV2ZWwuIiBUaGlzIGRpdmlkZXMgdGhlIHRyZWUgaW50byB0d28gbWFpbiBicmFuY2hlcyBvciBzdWJ0cmVlczoNCiAgICAtIFJpZ2h0IFN1YnRyZWU6IFRoaXMgY29tcHJpc2VzIGluc3RhbmNlcyB3aXRoIFNlbmlvciAoU0UpIGFuZCBFeGVjdXRpdmUgKEVYKSBleHBlcmllbmNlIGxldmVscy4NCiAgICAtIExlZnQgU3VidHJlZTogVGhpcyBpbmNsdWRlcyBpbmRpdmlkdWFscyB3aXRoIEVudHJ5IChFTikgYW5kIE1pZCAoTUkpIGV4cGVyaWVuY2UgbGV2ZWxzLg0KDQoyLiBXaXRoaW4gdGhlIHJpZ2h0IHN1YnRyZWU6DQogICAtIElmIHRoZSAnZXhwZXJpZW5jZSBsZXZlbCcgaXMgNCAoRVgsIEV4ZWN1dGl2ZSBsZXZlbCkgLCB0aGUgdHJlZSBzcGxpdHMgYmFzZWQgb24gdGhlICdFbXBsb3llZV9yZXNpZGVuY2UnIGF0dHJpYnV0ZS4gSXQgY2hlY2tzIHdoZXRoZXIgdGhlICdFbXBsb3llZV9yZXNpZGVuY2UnIGlzICdMYXRpbiBBbWVyaWNhLicNCiAgIC0gSWYgJ0VtcGxveWVlX3Jlc2lkZW5jZScgZG9lcyBub3QgZXF1YWwgJ0xhdGluIEFtZXJpY2EsJyB0aGUgZGlmZmVyZW50aWF0aW9uIGNvbnRpbnVlcyB3aXRoIHRoZSAncmVtb3RlX3JhdGlvJyBhdHRyaWJ1dGUsIGZ1cnRoZXIgZGl2aWRpbmcgdGhlIHRyZWUuDQoNCjMuIFdpdGhpbiB0aGUgbGVmdCBzdWJ0cmVlOg0KICAgLSBJZiB0aGUgJ2V4cGVyaWVuY2UgbGV2ZWwnIGlzIDEgKEVOLCBFbnRyeSBsZXZlbCksIHRoZSB0cmVlIGRpdmlkZSBiYXNlZCBvbiB0aGUgJ0VtcGxveWVlX3Jlc2lkZW5jZScgYXR0cmlidXRlLCBzcGVjaWZpY2FsbHkgY2hlY2tpbmcgZm9yICdTdWItU2FoYXJhbiBBZnJpY2EuJw0KICAgLSBJZiB0aGUgJ2V4cGVyaWVuY2UgbGV2ZWwnIGlzIDIgKE1JLCBNaWQgbGV2ZWwpLCBpdCBhbHNvIGJyYW5jaGVzIGJhc2VkIG9uICdFbXBsb3llZV9yZXNpZGVuY2UsJyBidXQgaW4gdGhpcyBjYXNlLCBsb29raW5nIHRvIHNlZSBpZiBpdCBlcXVhbHMgJ05vcnRoIEFtZXJpY2EuJw0KDQpUaGUgZGVjaXNpb24gdHJlZSBjb250aW51ZXMgdG8gc2VsZWN0IHRoZSBtb3N0IGFwcHJvcHJpYXRlIGF0dHJpYnV0ZXMgZm9yIHNwbGl0dGluZyBhdCBlYWNoIG5vZGUsIHByb2dyZXNzaXZlbHkgcmVmaW5pbmcgdGhlIGRlY2lzaW9uIHByb2Nlc3MgdW50aWwgaXQgcmVhY2hlcyB0aGUgbGVhdmVzLCB3aGVyZSBmaW5hbCBjbGFzcyBsYWJlbHMgYXJlIGFzc2lnbmVkIHRvIHRoZSBpbnN0YW5jZXMuDQoNCg0KDQojIyMjIyBTZW5zaXRpdml0eSwgQWNjdXJhY3ksIFNwZWNpZml0eSBhbmQgcHJlY2lzaW9uIG9mIGFsbCAzLDUgYW5kIDEwIGZvbGRzIHVzaW5nIEdhaW4gcmF0aW8NCg0KYGBge3J9DQpyYmluZCgiMTAgRm9sZHMiPW1hY3JvKGdhaW5SYXRpbzEwY20pLCAiNSBGb2xkcyI9bWFjcm8oZ2FpblJhdGlvNWNtKSwgIjMgRm9sZHMiPW1hY3JvKGdhaW5SYXRpbzNjbSkgICkgDQpgYGANCg0KQmFzZWQgb24gdGhlIGV2YWx1YXRpb24gbWV0cmljcyBvZiBhdmVyYWdlIFNlbnNpdGl2aXR5LFByZWNpc2lvbiAsU3BlY2lmaWNpdHksIGFuZCBBY2N1cmFjeSwgaXQgaXMgZXZpZGVudCB0aGF0IHRoZSBnYWluIHJhdGlvIG1vZGVsLCBidWlsdCB1c2luZyBhIDEwLWZvbGQgY3Jvc3MtdmFsaWRhdGlvbiBhcHByb2FjaCwgZXhoaWJpdHMgc3VwZXJpb3IgcGVyZm9ybWFuY2UgY29tcGFyZWQgdG8gdGhlIG90aGVyIHR3byBtb2RlbHMuIEhvd2V2ZXIsIGl0J3Mgd29ydGggbm90aW5nIHRoYXQgdGhlIGRpZmZlcmVuY2UgaW4gcGVyZm9ybWFuY2UgYmV0d2VlbiB0aGUgbW9kZWxzIGlzIHJlbGF0aXZlbHkgc21hbGwuDQoNCkEgZGV0YWlsZWQgZXhhbWluYXRpb24gb2YgdGhlIHJlc3VsdHMgZnJvbSB0aGUgMTAtZm9sZCBjcm9zcy12YWxpZGF0aW9uIHJldmVhbHMgdGhhdCB0aGUgbW9kZWwgaGFzIGEgbm90YWJseSBoaWdoIHNwZWNpZmljaXR5IGNvbXBhcmVkIHRvIG90aGVyIG1ldHJpY3MuIFRoaXMgaGlnaCBzcGVjaWZpY2l0eSBzdWdnZXN0cyB0aGF0IHRoZSBtb2RlbCBpcyBwYXJ0aWN1bGFybHkgZWZmZWN0aXZlIGF0IGNvcnJlY3RseSBpZGVudGlmeWluZyBpbnN0YW5jZXMgdGhhdCBkbyBub3QgcGVydGFpbiB0byB0aGUgdGFyZ2V0IGNsYXNz4oCUZXNzZW50aWFsbHksIGl0IGFjY3VyYXRlbHkgcmVjb2duaXplcyB3aGVuIGV4YW1wbGVzIGFyZSBub3QgbWVtYmVycyBvZiB0aGUgc3BlY2lmaWVkIGNsYXNzLiBGb3IgZXhhbXBsZSwgaWYgdGhlIHBvc2l0aXZlIGNsYXNzIGluIHF1ZXN0aW9uIGlzIOKAnEhpZ2jigJ0gdGhlbiB0aGUgbW9kZWwgaXMgYWJsZSB0byBjb3JyZWN0bHkgY2xhc3NpZnkgdHVwbGVzIHRoYXQgYmVsb25nIHRvIOKAnFZlcnkgTG934oCdLCDigJxNZWRpdW3igJ0sIGFuZCDigJxWZXJ5IEhpZ2jigJ0uDQoNCkhvd2V2ZXIsIHBvc3Nlc3NpbmcgaGlnaCBzcGVjaWZpY2l0eSBhbG9uZSBkb2VzIG5vdCBndWFyYW50ZWUgdGhlIG92ZXJhbGwgZWZmZWN0aXZlbmVzcyBvZiB0aGUgbW9kZWwsIGFzIGEgd2VsbC1yb3VuZGVkIG1vZGVsIGFsc28gcmVxdWlyZXMgYmFsYW5jZWQgcGVyZm9ybWFuY2UgYWNyb3NzIG90aGVyIG1ldHJpY3MuIEluIHRoaXMgY2FzZSwgaXRzIGFiaWxpdHkgdG8gY2FwdHVyZSBhbmQgY2xhc3NpZnkgaW5zdGFuY2VzIHRoYXQgZG8gYmVsb25nIHRvIHRoZSBwb3NpdGl2ZSBjbGFzcyAoYXMgbWVhc3VyZWQgYnkgc2Vuc2l0aXZpdHkpIGlzIG5vdCBhcyByb2J1c3QuIEZvciBhIG1vZGVsIHRvIGJlIGNvbnNpZGVyZWQgdHJ1bHkgZWZmZWN0aXZlLCBpdCB3b3VsZCBuZWVkIHRvIGRlbW9uc3RyYXRlIHN0cm9uZyBwZXJmb3JtYW5jZSBpbiBhbGwgbWV0cmljcyBzcGVjaWZpY2l0eSBhbmQgc2Vuc2l0aXZpdHksIGVuc3VyaW5nIGl0IGNhbiBhY2N1cmF0ZWx5IGRpc3Rpbmd1aXNoIGJvdGggbmVnYXRpdmUgYW5kIHBvc2l0aXZlIGluc3RhbmNlcyBhcyB3ZWxsIGFzIGFjY3VyYWN5IHByZWNpc2lvbi4NCg0KDQoNCg0KDQoNCiMjIyBJbmZvcm1hdGlvbiBnYWluDQoNCkluZm9ybWF0aW9uIEdhaW4gaXMgYSBtZXRyaWMgdXNlZCB0byBkZWNpZGUgd2hpY2ggYXR0cmlidXRlIHRvIGNob29zZSBmb3Igc3BsaXR0aW5nIHRoZSBkYXRhIGF0IGVhY2ggbm9kZSBpbiB0aGUgZGVjaXNpb24gdHJlZS4gRm9yIGEgZ2l2ZW4gZGF0YXNldCwgdGhlIEluZm9ybWF0aW9uIEdhaW4gb2YgYW4gYXR0cmlidXRlIGlzIGNhbGN1bGF0ZWQgYnkgY29tcGFyaW5nIHRoZSBlbnRyb3B5IGJlZm9yZSBhbmQgYWZ0ZXIgdGhlIGRhdGFzZXQgaXMgc3BsaXQgYmFzZWQgb24gdGhhdCBhdHRyaWJ1dGUuIFRoZSBhdHRyaWJ1dGUgd2l0aCB0aGUgaGlnaGVzdCBJbmZvcm1hdGlvbiBHYWluIGlzIGNob3NlbiBhcyB0aGUgc3BsaXR0aW5nIGF0dHJpYnV0ZS4NCg0KIyMjIyAxMCBGb2xkcw0KDQpUaGUgdHJlZSBvZiB0aGUgaW5mb3JtYXRpb24gZ2FpbiB1c2luZyAxMCBmb2xkcw0KDQpgYGB7ciwgZmlnLmhlaWdodD03MCwgZmlnLndpZHRoPTkwfQ0Kc2V0LnNlZWQoMTApDQpjdHJsIDwtIHRyYWluQ29udHJvbChtZXRob2QgPSAiY3YiLCBudW1iZXIgPSAxMCwgcmV0dXJuUmVzYW1wPSJhbGwiLCBzYXZlUHJlZGljdGlvbnM9ImZpbmFsIikNCg0KDQppbmZvR2FpbjEwIDwtIHRyYWluKHNhbGFyeV9pbl91c2QgfiAuLCBkYXRhID0gYmFsYW5jZWRfZGF0YXNldCwgbWV0aG9kID0gIkM1LjAiLHRyQ29udHJvbCA9IGN0cmwpDQoNCmM1bW9kZWwgPC0gQzUuMChzYWxhcnlfaW5fdXNkIH4gLiwNCiAgICAgICAgICAgICAgICAgICAgICAgZGF0YSA9IGJhbGFuY2VkX2RhdGFzZXQsDQogICAgICAgICAgICAgICAgICAgICAgIHRyaWFscyA9IGluZm9HYWluMTAkYmVzdFR1bmUkdHJpYWxzLCANCiAgICAgICAgICAgICAgICAgICAgICAgcnVsZXMgPSBGQUxTRSwNCiAgICAgICAgICAgICAgICAgICAgICAgY29udHJvbCA9IEM1LjBDb250cm9sKHdpbm5vdyA9IGluZm9HYWluMTAkYmVzdFR1bmUkd2lubm93KSkNCnBsb3QoYzVtb2RlbCkNCmBgYA0KDQpmcm9tIHRoZSB0cmVlLCB0aGUgImV4cGVyaW5jZSBsZXZlbCIgYXR0cmlidXRlIHdhcyB0aGUgZmlyc3Qgc2VsZWN0ZWQgc3BsaXR0aW5nIGF0dHJpYnV0ZSBtZWFuaW5nIHRoYXQgaXQgaGFzIHRoZSBoaWdoZXN0IGluZm9ybWF0aW9uIGdhaW4gYW1vbmcgYWxsIGF0dHJpYnV0ZXMNCg0KDQoNCg0KIyMjIyMgQ29uZnVzaW9uIG1hdHJpeCBvZiAxMCBmb2xkcyB1c2luZyBJbmZvcm1hdGlvbiBnYWluDQoNClRoZSBmb2xsb3dpbmcgY29uZnVzaW9uIE1hdHJpeCB3aWxsIHNob3cgdGhlIHBlcmZvcm1hbmNlIG9mIHRoZSBjbGFzc2lmaWVyIHVzaW5nIHRoZSBwcmVkaWN0ZWQgY2xhc3MgbGFiZWxzIGFuZCB0aGUgYWN0dWFsIGNsYXNzIGxhYmVscyBvZiBvdXIgZGF0YXNldA0KDQpgYGB7cn0NCmluZm9HYWluMTBjbT0gY2FyZXQ6OmNvbmZ1c2lvbk1hdHJpeChpbmZvR2FpbjEwJHByZWQkb2JzLCBpbmZvR2FpbjEwJHByZWQkcHJlZCkNCg0KaW5mb0dhaW4xMGNtDQoNCmBgYA0KDQpzaW1pbGFyIHRvIHRoZSB0cmVlcyBmcm9tIHRoZSBnaW5pIGluZGV4IGFuZCBnYWluIHJhdGlvLCB0aGUgY2xhc3NpZmllciBzZWVtIHRvIGhhdmUgYmV0dGVyIHBlcmZvcm1hbmNlIHdoZW4gdHJlYXRpbmcgdGhlICJ2ZXJ5IGhpZ2giIGNsYXNzIGFzIHRoZSBwb3NpdGl2ZSBjbGFzcw0KDQoNCg0KDQoNCiMjIyMgNSBGb2xkcw0KDQpUaGUgdHJlZSBvZiB0aGUgaW5mb3JtYXRpb24gZ2FpbiB1c2luZyA1IGZvbGRzDQoNCmBgYHtyLCBmaWcuaGVpZ2h0PTcwLCBmaWcud2lkdGg9OTB9DQpzZXQuc2VlZCgxMCkNCmN0cmwgPC0gdHJhaW5Db250cm9sKG1ldGhvZCA9ICJjdiIsIG51bWJlciA9IDUsIHJldHVyblJlc2FtcD0iYWxsIiwgc2F2ZVByZWRpY3Rpb25zPSJmaW5hbCIpDQoNCg0KaW5mb0dhaW41IDwtIHRyYWluKHNhbGFyeV9pbl91c2QgfiAuLCBkYXRhID0gYmFsYW5jZWRfZGF0YXNldCwgbWV0aG9kID0gIkM1LjAiLHRyQ29udHJvbCA9IGN0cmwpDQoNCmM1bW9kZWwgPC0gQzUuMChzYWxhcnlfaW5fdXNkIH4gLiwNCiAgICAgICAgICAgICAgICAgICAgICAgZGF0YSA9IGJhbGFuY2VkX2RhdGFzZXQsDQogICAgICAgICAgICAgICAgICAgICAgIHRyaWFscyA9IGluZm9HYWluNSRiZXN0VHVuZSR0cmlhbHMsIA0KICAgICAgICAgICAgICAgICAgICAgICBydWxlcyA9IEZBTFNFLA0KICAgICAgICAgICAgICAgICAgICAgICBjb250cm9sID0gQzUuMENvbnRyb2wod2lubm93ID0gaW5mb0dhaW41JGJlc3RUdW5lJHdpbm5vdykpDQpwbG90KGM1bW9kZWwpDQpgYGANCg0KdGhlIHRyZWUgaGFzIHNpbWlsYXIgYmVoYXZpb3IgYXMgdGhlIDEwIGZvbGRzIGluZm9ybWF0aW9uIGdhaW4gdHJlZQ0KDQoNCg0KDQoNCg0KIyMjIyMgQ29uZnVzaW9uIG1hdHJpeCBvZiA1IGZvbGRzIHVzaW5nIEluZm9ybWF0aW9uIGdhaW4NCg0KVGhlIGZvbGxvd2luZyBjb25mdXNpb24gTWF0cml4IHdpbGwgc2hvdyB0aGUgcGVyZm9ybWFuY2Ugb2YgdGhlIGNsYXNzaWZpZXIgdXNpbmcgdGhlIHByZWRpY3RlZCBjbGFzcyBsYWJlbHMgYW5kIHRoZSBhY3R1YWwgY2xhc3MgbGFiZWxzIG9mIG91ciBkYXRhc2V0DQoNCmBgYHtyfQ0KaW5mb0dhaW41Y20gPSBjYXJldDo6Y29uZnVzaW9uTWF0cml4KGluZm9HYWluNSRwcmVkJG9icywgaW5mb0dhaW41JHByZWQkcHJlZCkNCg0KaW5mb0dhaW41Y20NCg0KYGBgDQoNCnRoZSBjbGFzc2lmaWVyIHNob3dzIHZlcnkgY2xvc2UgcGVyZm9ybWFuY2UgdG8gdGhlIDEwIGZvbGRzIGluZm9ybWF0aW9uIGdhaW4gbW9kZWwNCg0KDQoNCg0KDQoNCiMjIyMgMyBGb2xkcw0KDQpUaGUgdHJlZSBvZiB0aGUgaW5mb3JtYXRpb24gZ2FpbiB1c2luZyAzIGZvbGRzDQoNCmBgYHtyLCBmaWcuaGVpZ2h0PTcwLCBmaWcud2lkdGg9OTB9DQpzZXQuc2VlZCgxMCkNCmN0cmwgPC0gdHJhaW5Db250cm9sKG1ldGhvZCA9ICJjdiIsIG51bWJlciA9IDMsIHJldHVyblJlc2FtcD0iYWxsIiwgc2F2ZVByZWRpY3Rpb25zPSJmaW5hbCIpDQoNCg0KaW5mb0dhaW4zIDwtIHRyYWluKHNhbGFyeV9pbl91c2QgfiAuLCBkYXRhID0gYmFsYW5jZWRfZGF0YXNldCwgbWV0aG9kID0gIkM1LjAiLHRyQ29udHJvbCA9IGN0cmwpDQoNCmM1bW9kZWwgPC0gQzUuMChzYWxhcnlfaW5fdXNkIH4gLiwNCiAgICAgICAgICAgICAgICAgICAgICAgZGF0YSA9IGJhbGFuY2VkX2RhdGFzZXQsDQogICAgICAgICAgICAgICAgICAgICAgIHRyaWFscyA9IGluZm9HYWluMyRiZXN0VHVuZSR0cmlhbHMsIA0KICAgICAgICAgICAgICAgICAgICAgICBydWxlcyA9IEZBTFNFLA0KICAgICAgICAgICAgICAgICAgICAgICBjb250cm9sID0gQzUuMENvbnRyb2wod2lubm93ID0gaW5mb0dhaW4zJGJlc3RUdW5lJHdpbm5vdykpDQpwbG90KGM1bW9kZWwpDQpgYGANCg0KdGhlIHZpc2lzYmxlIHBhcnRzIG9mIHRoZSB0cmVlIHNlZW0gdG8gYmVoYXZlIHRoZSBzYW1lIGFzIHRoZSBwcmV2b2l1cyAyIGZvbGQgc2l6ZXMtIDEwIGFuZCA1Lg0KDQoNCg0KDQojIyMjIyBDb25mdXNpb24gbWF0cml4IG9mIDMgZm9sZHMgdXNpbmcgSW5mb3JtYXRpb24gZ2Fpbg0KDQpUaGUgZm9sbG93aW5nIGNvbmZ1c2lvbiBNYXRyaXggd2lsbCBzaG93IHRoZSBwZXJmb3JtYW5jZSBvZiB0aGUgY2xhc3NpZmllciB1c2luZyB0aGUgcHJlZGljdGVkIGNsYXNzIGxhYmVscyBhbmQgdGhlIGFjdHVhbCBjbGFzcyBsYWJlbHMgb2Ygb3VyIGRhdGFzZXQNCg0KYGBge3J9DQppbmZvR2FpbjNjbSA9IGNhcmV0Ojpjb25mdXNpb25NYXRyaXgoaW5mb0dhaW4zJHByZWQkb2JzLCBpbmZvR2FpbjMkcHJlZCRwcmVkKQ0KDQppbmZvR2FpbjNjbQ0KDQpgYGANCg0Kc2luY2UgdGhlIHRyZWUgaXMgZXNzZW50aWFsbHkgc2ltaWxhciB0byB0aGUgcHJldmlvdXMgdHdvIGluZm9ybWF0aW9uIGdhaW4gdHJlZXMgdGhlIHJlc3VsdHMgdGhhdCB0aGlzIHRyZWUgc2hvd3MgaXMgdmVyeSBjbG9zZSBpbiBwZXJmb3JtYW5jZSB0byB0aGVtIGFzIHdlbGwuDQoNCg0KDQoNCg0KDQojIyMjIyBBbmFseXNpcyBvZiB0aGUgaW5mb3JtYXRpb24gZ2FpbiBjbGFzc2lmaWNhdGlvbg0KVGhlIG9ic2VydmVkIHN0cnVjdHVyZSBvZiB0aGUgdGhyZWUgZGVjaXNpb24gdHJlZXMgc2VlbXMgdG8gYmUgdGhlIHNhbWUgYW5kIGl0IGNhbiBiZSBzdW1tYXJpemVkIGFzIGZvbGxvd3M6DQoNCjEuIFJvb3QgTm9kZSAtIEV4cGVyaWVuY2UgTGV2ZWw6IFRoZSBpbml0aWFsIGF0dHJpYnV0ZSB1c2VkIGZvciBzcGxpdHRpbmcgdGhlIGRhdGFzZXQgYXQgdGhlIHJvb3Qgbm9kZSBpcyB0aGUgImV4cGVyaWVuY2UgbGV2ZWwuIiBUaGlzIGRpdmlkZXMgdGhlIHRyZWUgaW50byB0d28gbWFpbiBicmFuY2hlcyBvciBzdWJ0cmVlczoNCiAgICAtIFJpZ2h0IFN1YnRyZWU6IFRoaXMgY29tcHJpc2VzIGluc3RhbmNlcyB3aXRoIFNlbmlvciAoU0UpIGFuZCBFeGVjdXRpdmUgKEVYKSBleHBlcmllbmNlIGxldmVscy4NCiAgICAtIExlZnQgU3VidHJlZTogVGhpcyBpbmNsdWRlcyBpbmRpdmlkdWFscyB3aXRoIEVudHJ5IChFTikgYW5kIE1pZCAoTUkpIGV4cGVyaWVuY2UgbGV2ZWxzLg0KDQoyLiBXaXRoaW4gdGhlIHJpZ2h0IHN1YnRyZWU6DQpJbiB0aGUgcmlnaHQgc3ViIHRyZWUgaWYgdGhlIGV4cGVyaWVuY2UgbGV2ZWwgaXMgNChFWCkgdGhlIHRyZWUgd2lsbCBiZSBkaXZpZGVkIGJhc2VkIG9uIOKAnENvbXBhbnkgbG9jYXRpb27igJ0NCg0KMy4gV2l0aGluIHRoZSBsZWZ0IHN1YnRyZWU6DQpJbiB0aGUgbGVmdCBzdWJ0cmVlIGl0IHdpbGwgZGl2aWRlIHRoZSB0cmVlIGZvciBib3RoIHR3byBleHBlcmllbmNlIGxldmVscyAxKEVOKSBhbmQgMihNSSkgYmFzZWQgb24g4oCcZW1wbG95ZWUgcmVzaWRlbmNl4oCdDQphbmQgd2hlbiB0aGUg4oCcZW1wbG95ZWUgcmVzaWRlbmNl4oCdIGlzIOKAnE5vcnRoIEFtZXJpY2HigJ0gdGhlIHRyZWUgd2lsbCBiZSBmdXJ0aGVyIGRpdmlkZWQgYmFzZWQgb24g4oCcc2FsYXJ5IGN1cnJlbmN54oCdIGFuZCB3aGVuIHRoaXMgYXR0cmlidXRlIGlzIGVxdWFsIHRvIOKAnFVTROKAnSB0aGUgZGl2aXNpb24gd2lsbCBiZSBiYXNlZCBvbiB0aGUg4oCcam9iIHRpdGxl4oCdIGF0dHJpYnV0ZQ0KDQpUaGUgZGVjaXNpb24gdHJlZSBjb250aW51ZXMgdG8gc2VsZWN0IHRoZSBtb3N0IGFwcHJvcHJpYXRlIGF0dHJpYnV0ZXMgZm9yIHNwbGl0dGluZyBhdCBlYWNoIG5vZGUsIHByb2dyZXNzaXZlbHkgcmVmaW5pbmcgdGhlIGRlY2lzaW9uIHByb2Nlc3MgdW50aWwgaXQgcmVhY2hlcyB0aGUgbGVhdmVzLCB3aGVyZSBmaW5hbCBjbGFzcyBsYWJlbHMgYXJlIGFzc2lnbmVkIHRvIHRoZSBpbnN0YW5jZXMuDQoNCg0KIyMjIyMgU2Vuc2l0aXZpdHksIEFjY3VyYWN5LCBTcGVjaWZpdHkgYW5kIHByZWNpc2lvbiBvZiBhbGwgMyw1IGFuZCAxMCBmb2xkcyB1c2luZyBJbmZvcm1hdGlvbiBnYWluDQoNCmBgYHtyfQ0KcmJpbmQoIjEwIEZvbGRzIj1tYWNybyhpbmZvR2FpbjEwY20pLCAiNSBGb2xkcyI9bWFjcm8oaW5mb0dhaW41Y20pLCAiMyBGb2xkcyI9bWFjcm8oaW5mb0dhaW4zY20pICApIA0KYGBgDQpCYXNlZCBvbiB0aGUgcHJvdmlkZWQgc2Vuc2l0aXZpdHksIHNwZWNpZmljaXR5LCBwcmVjaXNpb24sIGFuZCBhY2N1cmFjeSB2YWx1ZXMgdGhlcmUgaXNuJ3QgYSBjbGVhciBpbmRpY2F0aW9uIG9mIHRoZSBzdXBlcmlvcml0eSBvZiBvbmUgZm9sZCBvdmVyIGFub3RoZXIgZm9yIEluZm9ybWF0aW9uIEdhaW4gbW9kZWwgLndlIG1heSBuZWVkIHRvIGNvbnNpZGVyIGFkZGl0aW9uYWwgZmFjdG9ycyBvciBjb25kdWN0IGZ1cnRoZXIgYW5hbHlzaXMgdG8gbWFrZSBhIHdlbGwtaW5mb3JtZWQgZGVjaXNpb24uIGFzIGNhbiBiZSBzZWVuIGluIHRoZSB0YWJsZSB0aGUgMTAgZm9sZHMgaGFzIHRoZSBiZXN0IFNwZWNpZmljaXR5IGFuZCBQcmVjaXNpb24sIG1lYW53aGlsZSB0aGUgNSBmb2xkcyBoYXMgdGhlIGJlc3QgU2Vuc2l0aXZpdHkgYW5kIEFjY3VyYWN5Lg0KDQoNCg0KDQojIyMgQW5hbGFzeXMgb2YgdGhlIHRocmVlIGNsYXNzaWZpY2F0aW9uIG1ldGhvZHMNCg0KVGhlIGdhaW4gcmF0aW8gZXZhbHVhdGVkIHdpdGggMTAtZm9sZCBjcm9zcy12YWxpZGF0aW9uIGFwcGVhcnMgdG8gaGFzIHRoZSBiZXN0IHBlcmZvcm1hbmNlIGFtb25nIGFsbCB0aGUgZGVjaXNpb24gdHJlZSBtb2RlbHMuIFRoaXMgbWlnaHQgYmUgZHVlIHRvIHRoZSBmYWN0IHRoYXQgdGhlIGdhaW4gcmF0aW8gdGVuZHMgdG8gZmF2b3IgdW5iYWxhbmNlZCBzcGxpdHMsIHdoZXJlIG9uZSBwYXJ0aXRpb24gaXMgc2lnbmlmaWNhbnRseSBzbWFsbGVyIHRoYW4gb3RoZXJzLiBJbiBvdXIgZGF0YXNldCwgaWYgYW4gYXR0cmlidXRlIGhhcyBhIHJhcmUgdmFsdWUsIHRoZSBnYWluIHJhdGlvIG1pZ2h0IHByaW9yaXRpemUgc3BsaXR0aW5nIG9uIHRoaXMgYXR0cmlidXRlIGRlc3BpdGUgdGhlIHJlc3VsdGluZyB1bmJhbGFuY2UuIA0KDQpEZXNwaXRlIHRoZSBzdXBlcmlvcml0eSBvZiB0aGUgMTAtZm9sZCwgdGhlIHBlcmZvcm1hbmNlIG1ldHJpY3Mgb2YgYWxsIHRocmVlIGZvbGQgc2l6ZXMgKDMtZm9sZCwgNS1mb2xkLCBhbmQgMTAtZm9sZCkgdXNpbmcgR2luaSBpbmRleCwgZ2FpbiByYXRpbywgYW5kIGluZm9ybWF0aW9uIGdhaW4gYXJlIHJlbGF0aXZlbHkgc2ltaWxhciwgc3VnZ2VzdGluZyB0aGF0IGFsbCB0aHJlZSBtZWFzdXJlcyBhcmUgcm9idXN0IHdpdGhpbiB0aGUgY29udGV4dCBvZiB0aGlzIGRhdGFzZXQuIEEgbGlrZWx5IGNvbnRyaWJ1dGluZyBmYWN0b3IgdG8gdGhpcyBwZXJmb3JtYW5jZSBjb25zaXN0ZW5jeSBpcyB0aGUgYmFsYW5jZWQgZGlzdHJpYnV0aW9uIG9mIGNsYXNzIGxhYmVscyBpbiB0aGUgZGF0YXNldC4gV2hlbiBjbGFzc2VzIGFyZSBiYWxhbmNlZCwgZWFjaCBzcGxpdHRpbmcgY3JpdGVyaW9uIGlzIG1vcmUgb3IgbGVzcyBlcXVhbGx5IGxpa2VseSB0byBlbmNvdW50ZXIgaW5mb3JtYXRpdmUgc3BsaXRzLCB3aGljaCBzZXJ2ZXMgdG8gZGVjcmVhc2UgcGVyZm9ybWFuY2UgdmFyaWFiaWxpdHkgYWNyb3NzIGRpZmZlcmVudCBzcGxpdHRpbmcgbWV0aG9kcy4NCg0KDQoNCiMjIENsdXN0ZXJpbmcNCg0KRGF0YSBjbHVzdGVyaW5nIGlzIGEgcHJvY2VzcyB0byBwYXJ0aXRpb24gZGF0YSBpbnRvIGdyb3VwcyBvciBjbHVzdGVycyxpdCBpcyBhbiB1bnN1cGVydmlzZWQgbGVhcm5pbmcgcHJvY2Vzcywgd2hpY2ggaXMgZXhjdXRlZCB3aXRob3V0IGtub3dpbmcgdGhlIGNsYXNzIGxhYmVsIG9mIHRoZSB0cmFpbmluZyBkYXRhLiBUaGUgZGF0YSBpbiB0aGUgc2FtZSBncm91cCAiY2x1c3RlciIgYXJlIHNpbWlsYXIgdG8gb25lIGFub3RoZXIgYW5kIGRpZmZlcmVudCBmcm9tIGRhdGEgaW4gb3RoZXIgY2x1c3RlcnMuIEFuZCBmb3IgdGhpcyBkYXRhIG1pbmluZyB0YXNrIFdlIHdpbGwgdXRpbGl6ZSBrLW1lYW5zIGNsdXN0ZXJpbmcuDQoNCiMjIyAxLSBwcmVwcmVvY2Vzc2luZw0KDQp3ZSB3aWxsIGVuY29kZSB0aGUgcmVzdCBvZiBmYWN0b3IgY29sdW1ucyB0byB0cmFuc2Zvcm0gdGhlbSBpbnRvIG51bWVyaWMgdHlwZXMgYmVmb3JlIGNsdXN0ZXJpbmcsIGVuYWJsaW5nIG1lYW5pbmdmdWwgZGlzdGFuY2UgY2FsY3VsYXRpb25zIHVzaW5nIGttZWFucyBhbmQgb3RoZXIgZm9ybXVsYXMsIGFuZCBhbGxvd2luZyBmb3IgbWF4aW11bSBmbGV4aWJpbGl0eSBpbiBkYXRhIHByb2Nlc3NpbmcgYW5kIGludGVycHJldGF0aW9uLiB3ZSB3aWxsIGFsc28gcmVtb3ZlIHRoZSBjbGFzcyBsYWJlbCBmcm9tIHRoZSBkYXRhc2V0IGFzIGNsdXN0ZXJpbmcgaXMgYW4gdW5zdXBlcnZpc2VkIGxlYXJuaW5nIHByb2Nlc3MsIGFuZCB3ZSB3aWxsIHByZXNlcnZlIHRoaXMgY2xhc3MgbGFiZWwgaW4gYW4gYXR0cmlidXRlIGZvciBsYXRlciB1c2UuIGxhc3RseSwgd2Ugd2lsbCBzY2FsZSBhbGwgbnVtZXJpYyBhdHRyaWJ1dGVzIGluIHRoZSBkYXRhc2V0IHNvIHRoZXkgd2lsbCBiZSBzdGFuZGFyaXplZC4NCg0KYGBge3J9DQoNCiMgdmlldyBkYXRhDQoNCmRhdGFzZXQzIDwtIGRhdGFzZXQyDQpWaWV3KGRhdGFzZXQzKQ0KDQojIFJlc2VydmUgdGhlIHNhbGFyeV9pbl91c2QgKHRoZSBjbGFzcyBsYWJlbCkgY29sdW1uIGluIGFuIGF0dHJpYnV0ZSBiZWZvcmUgcmVtb3ZpbmcgaXQgZnJvbSB0aGUgZGF0YXNldCBmb3IgY2x1c3RlcmluZw0KDQpjbGFzc0xhYmVsIDwtIGRhdGFzZXQzWywgNV0gDQoNCg0KIyBSZW1vdmUgdGhlIGNsYXNzIGxhYmxlIGZyb20gdGhlIGRhdGFzZXQNCg0KZGF0YXNldDMgPC0gZGF0YXNldDNbLCAtNV0NCg0KIyBlbmNvZGluZyBqb2JfdGl0bGUgdmFyaWFibGUNCg0KZGF0YXNldDMkam9iX3RpdGxlID0gZmFjdG9yKGRhdGFzZXQzJGpvYl90aXRsZSwgbGV2ZWxzPWMoIkFuYWx5c3QiLCAiQXJjaGl0ZWN0IiwgIkVuZ2luZWVyIiwgIkxlYWRlcnNoaXAiLCAiQ29uc3VsdGFudC9TcGVjaWFsaXN0IiwiQ3liZXIgU2VjdXJpdHkiLCJPdGhlcnMiICksIGxhYmVscz1jKDQsMSwyLDUsMyw2LDcpKQ0KDQojIGVuY29kaW5nIHNhbGFyeV9jdXJyZW5jeSB2YXJpYWJsZQ0KDQpkYXRhc2V0MyRzYWxhcnlfY3VycmVuY3kgPSBmYWN0b3IoZGF0YXNldDMkc2FsYXJ5X2N1cnJlbmN5LCBsZXZlbHM9YygiVVNEIiwiQlJMIiwiR0JQIiwiRVVSIiwiSU5SIiwiQ0FEIiwiQ0hGIiwiREtLIiwiU0dEIiwiQVVEIiwiU0VLIiwiTVhOIiwiSUxTIiwiUExOIiwiTk9LIiwiSURSIiwiTlpEIiwiSFVGIiwiWkFSIiwiVFdEIiwiUlVCIiksIGxhYmVscz1jKDEsMiwzLDQsNSw2LDcsOCw5LDEwLDExLDEyLDEzLDE0LDE1LDE2LDE3LDE4LDE5LDIwLDIxKSkNCg0KIyBlbmNvZGluZyBlbXBsb3llZV9yZXNpZGVuY2UgdmFyaWFibGUNCg0KZGF0YXNldDMkZW1wbG95ZWVfcmVzaWRlbmNlID0gZmFjdG9yKGRhdGFzZXQzJGVtcGxveWVlX3Jlc2lkZW5jZSwgbGV2ZWxzPWMoIk5vcnRoIEFtZXJpY2EiLCJMYXRpbiBBbWVyaWNhICYgQ2FyaWJiZWFuIiwiU3ViLVNhaGFyYW4gQWZyaWNhIiwgIkV1cm9wZSAmIENlbnRyYWwgQXNpYSIsIkVhc3QgQXNpYSAmIFBhY2lmaWMiLCJTb3V0aCBBc2lhIiwiTWlkZGxlIEVhc3QgJiBOb3J0aCBBZnJpY2EiKSwgbGFiZWxzPWMoMSwyLDMsNCw1LDYsNykpDQoNCiMgZW5jb2RpbmcgY29tcGFueV9sb2NhdGlvbiB2YXJpYWJsZQ0KDQpkYXRhc2V0MyRjb21wYW55X2xvY2F0aW9uID0gZmFjdG9yKGRhdGFzZXQzJGNvbXBhbnlfbG9jYXRpb24sIGxldmVscz1jKCJOb3J0aCBBbWVyaWNhIiwiTGF0aW4gQW1lcmljYSAmIENhcmliYmVhbiIsIlN1Yi1TYWhhcmFuIEFmcmljYSIsICJFdXJvcGUgJiBDZW50cmFsIEFzaWEiLCJFYXN0IEFzaWEgJiBQYWNpZmljIiwiU291dGggQXNpYSIsIk1pZGRsZSBFYXN0ICYgTm9ydGggQWZyaWNhIiwgIkFRIiwgIlVNIiksIGxhYmVscz1jKDEsMiwzLDQsNSw2LDcsOCw5KSkNCg0KDQogDQojRGF0YSB0eXBlcyB0byBiZSB0cmFuc2Zvcm1lZCBpbnRvIG51bWVyaWMgdHlwZXMgYmVmb3JlIGNsdXN0ZXJpbmcNCiNUcmFuc2Zvcm1pbmcgYWxsIG5vbi1udW1lcmljIGF0dHJpYnV0ZXMgdG8gbnVtZXJpYyB0eXBlDQoNCg0KZGF0YXNldDMkZXhwZXJpZW5jZV9sZXZlbCA8LSBhcy5udW1lcmljKGFzLmNoYXJhY3RlcihkYXRhc2V0MyRleHBlcmllbmNlX2xldmVsKSkNCg0KZGF0YXNldDMkam9iX3RpdGxlIDwtIGFzLm51bWVyaWMoYXMuY2hhcmFjdGVyKGRhdGFzZXQzJGpvYl90aXRsZSkpDQoNCmRhdGFzZXQzJHNhbGFyeV9jdXJyZW5jeSA8LSBhcy5udW1lcmljKGFzLmNoYXJhY3RlcihkYXRhc2V0MyRzYWxhcnlfY3VycmVuY3kpKQ0KDQpkYXRhc2V0MyRlbXBsb3llZV9yZXNpZGVuY2UgPC0gYXMubnVtZXJpYyhhcy5jaGFyYWN0ZXIoZGF0YXNldDMkZW1wbG95ZWVfcmVzaWRlbmNlKSkNCg0KZGF0YXNldDMkY29tcGFueV9sb2NhdGlvbiA8LSBhcy5udW1lcmljKGFzLmNoYXJhY3RlcihkYXRhc2V0MyRjb21wYW55X2xvY2F0aW9uKSkNCg0KZGF0YXNldDMkY29tcGFueV9zaXplIDwtIGFzLm51bWVyaWMoYXMuY2hhcmFjdGVyKGRhdGFzZXQzJGNvbXBhbnlfc2l6ZSkpDQoNCiMgdml3ZSB0aGUgY2xhc3Mgb2YgYXR0cmlidXRlcyB0byBlbnN1cmUgdGhleSBoYXZlIHRyYW5zZm9ybWVkIHRvIG51bWVyaWMNCnNhcHBseShkYXRhc2V0MywgY2xhc3MpDQoNCg0KI3NjYWxlIGFsbCBhdHRyaWJ1dGVzIGluIHRoZSBkYXRhc2V0IHNvIHRoZXkgd291bGQgYmUgc3RhbmRhcmRpemVkIA0KZGF0YXNldDMgPC0gc2NhbGUoZGF0YXNldDMpDQoNCmBgYA0KDQoNCiMjIyAyLSBLLW1lYW5zDQoNCkFmdGVyIHByZXByb2Nlc3NpbmcgdGhlIGRhdGEsIG5vdyB3ZSBhcmUgcmVhZHkgdG8gcGVyZm9ybSB0aGUgY2x1c3RlcmluZyBwcm9jZXNzLCB3ZSB3aWxsIHVzZSB0aGUgay1tZWFucyBjbHVzdGVyaW5nLCBpdCBpcyBhIGNsdXN0ZXJpbmcgbWV0aG9kIHRoYXQgYWltcyB0byBtaW5pbWl6ZSB0aGUgc3VtIG9mIHNxdWFyZWQgZGlzdGFuY2VzIGJldHdlZW4gZWFjaCBkYXRhIHBvaW50IGFuZCB0aGUgY2VudHJvaWQgb2YgaXRzIGFzc2lnbmVkIGNsdXN0ZXIgYnkgaXRlcmF0aXZlbHkgdXBkYXRpbmcgY2x1c3RlciBhc3NpZ25tZW50cyBhbmQgY2VudHJvaWRzLg0KDQoNCiMjIyAzLSBDaG9vc2luZyBudW1iZXIgb2YgY2x1c3RlcnMgKGspDQoNCldlIHdpbGwgY2hvb3NlIDMgZGlmZmVyZW50IG51bWJlcnMgdG8gcGVyZm9ybSB0aGUgay1tZWFucyBjbHVzdGVyaW5nIG9uLCBvbmUgb2YgdGhlIG51bWJlcnMgc2hvdWxkIGJlIHJlbGF0ZXZpbHkgbGFyZ2UsIHRoZSBzZWNvbmQgc2hvdWxkIGJlIGluIHRoZSBtaWRkbGUgYW5kIHRoZSBsYXN0IHNob3VsZCBiZSBzbWFsbC4gVGhpcyB3YXkgd2Ugd2lsbCBjb3ZlciB0aGUgcG9zc2libGUgb3V0Y29tZXMgYW5kIGNsdXN0ZXJpbmcgcmVzdWx0cy4NCg0KIyMjIyBhLSBTaWxob3VldHRlIG1ldGhvZA0KDQpOb3cgd2Ugd2lsbCBhcHBseSBTaWxob3VldHRlIG1ldGhvZCB0byBmaW5kIHRoZSBvcHRpbWFsIG51bWJlciBvZiBjbHVzdGVycyBrLCB3ZSB3aWxsIGFsc28gcGxvdCBhIGdyYXBoIHdoZXJlIHgtYXhpcyByZXByZXNlbnQgdGhlIG51bWJlciBvZiBjbHVzdGVycyBhbmQgeS1heGlzIHJlcHJlc2VudCB0aGUgYXZlcmFnZSBTaWxob3VldHRlIGNvZWZmaWNpZW50DQoNCmBgYHtyfQ0KDQpmdml6X25iY2x1c3QoZGF0YXNldDMsIGttZWFucywgbWV0aG9kID0gInNpbGhvdWV0dGUiKSsNCiAgbGFicyhzdWJ0aXRsZSA9ICJTaWxob3VldHRlIG1ldGhvZCIpDQoNCmBgYA0KDQphcyBzZWVuIGJ5IHRoZSBncmFwaCwgdGhlIG51bWJlciBvZiBjbHVzdGVycyBrIHRoYXQgbWF4aW1pemVzIHRoZSBhdmVyYWdlIFNpbGhvdWV0dGUgY29lZmZpY2llbnQgaXMgMiwgc28gd2Ugd2lsbCB1c2UgaXQgZm9yIGNsdXN0ZXJpbmcuDQoNCiMjIyMgYi0gRWxib3cgbWV0aG9kDQoNClRoaXMgbWV0aG9kIGRldGVybWluZXMgdGhlIG51bWJlciBvZiBjbHVzdGVycyBhY2NvcmRpbmcgdG8gdGhlIHR1cm5pbmcgcG9pbnQgaW4gYSBjdXJ2ZSwgdGhlIGN1cnZlIGlzIHBsb3R0ZWQgdXNpbmcgdGhlIHRvdGFsIHdpdGhpbi1jbHVzdGVyIHN1bSBvZiBzcXVhcmUgKFdTUykgYXMgaW4geS1heGlzICwgYW5kIE5vLiBjbHVzdGVycyBpbiB4LWF4aXMNCg0KYGBge3J9DQoNCmZ2aXpfbmJjbHVzdChkYXRhc2V0Mywga21lYW5zLCBtZXRob2QgPSAid3NzIikgKw0KICBnZW9tX3ZsaW5lKHhpbnRlcmNlcHQgPSA0LCBsaW5ldHlwZSA9IDIpKw0KICBsYWJzKHN1YnRpdGxlID0gIkVsYm93IG1ldGhvZCIpDQoNCmBgYA0KDQpBcyBzaG93biwgdGhlIG51bWJlciBvZiBjbHVzdGVycyBrIHRoYXQgcmVwcmVzZW50cyB0aGUgdHVybmluZyBwb2ludCBpbiB0aGUgY3VydmUgaXMgNCwgc28gd2Ugd2lsbCB1c2UgaXQgZm9yIGNsdXN0ZXJpbmcuDQoNCkxhc3RseSwgd2Ugd2lsbCB1c2Ugaz0zIHNpbmNlIGl0IGFjaGVpdmVzIHRoZSBzZWNvbmQgaGlnaGVzdCBhdmVyYWdlIFNpbGhvdWV0dGUgY29lZmZpY2llbnQsIGFuZCBzaW5jZSBpdCdzIGluIHRoZSBtaWRkbGUgYmV0d2VlbiAyIGFuZCA0IGl0IHdpbGwgc3RyaWtlIGEgYmFsYW5jZSBiZXR3ZWVuIGhhdmluZyB0b28gZmV3IGNsdXN0ZXJzIChrPTIpLCBhbmQgaGF2aW5nIHNldmVyYWwgY2x1c3RlcnMgKGs9NCksIFRodXMsIHRoaXMgY2hvaWNlIHdpbGwgaGF2ZSBhbiBhY2NlcHRhYmxlIGFjdXVyYWN5Lg0KDQoNCiMjIyBrLW1lYW5zIGNsdXN0ZXJpbmcsIHZpc3VhbGl6YXRpb24gYW5kIGV2YWx1YXRpb24NCkluIHRoaXMgc2VjdGlvbiwgd2Ugd2lsbCBwZXJmb3JtIGstbWVhbnMgY2x1c3RlcmluZyBhbmQgdmlzdWFsaXplIGl0cyByZXN1bHQgdXNpbmcgdGhyZWUgZGlmZmVyZW50IGsncyB0aGF0IGhhdmUgYmVlbiBjaG9zZW4gYmVmb3JlaGFuZCwgdGhlbiB3ZSB3aWxsIGNvbXB1dGUgV1NTIGFuZCBCY3ViZWQgcHJlY2Vpc2lvbiBhbmQgcmVjYWxsIGFuZCBhdmVyYWdlIHNpbGhvdWV0dGUgZm9yIGVhY2ggY2x1c3RlciBhcyBtZXRob2RzIG9mIGV2YWx1YXRpbmcgY2x1c3RlcmluZyByZXN1bHRzLg0KDQojIyMjIGs9Mg0KDQpgYGB7cn0NCg0KI1VzZSBzZWVkIHRvIGd1YXJhbnRlZSByZXBsaWNhYmlsaXR5IG9mIHJhbmRvbSBwcm9jZXNzZXMNCnNldC5zZWVkKDg5NTMpDQoNCiMgcnVuIGstbWVhbnMgY2x1c3RlcmluZyB0byBmaW5kIDIgY2x1c3RlcnMNCmttZWFucy5yZXN1bHQgPC0ga21lYW5zKGRhdGFzZXQzLCAyKQ0KDQojIHByaW50IHRoZSBjbHVzdGVybmcgcmVzdWx0DQprbWVhbnMucmVzdWx0DQoNCiMgdmlzdWFsaXplIGNsdXN0ZXJpbmcNCmZ2aXpfY2x1c3RlcihrbWVhbnMucmVzdWx0LCBkYXRhID0gZGF0YXNldDMpDQoNCg0KI2F2ZXJhZ2Ugc2lsaG91ZXR0ZSBmb3IgZWFjaCBjbHVzdGVycw0KYXZnX3NpbCA8LSBzaWxob3VldHRlKGttZWFucy5yZXN1bHQkY2x1c3RlcixkaXN0KGRhdGFzZXQzKSkgDQpmdml6X3NpbGhvdWV0dGUoYXZnX3NpbCkNCg0KI1dpdGhpbi1jbHVzdGVyIHN1bSBvZiBzcXVhcmVzIHdzcyANCndzcyA8LSBrbWVhbnMucmVzdWx0JHRvdC53aXRoaW5zcw0KcHJpbnQod3NzKQ0KDQojQkN1YmVkDQprbWVhbnNfY2x1c3RlciA8LSBjKGttZWFucy5yZXN1bHQkY2x1c3RlcikNCg0KZ3JvdW5kX3RydXRoIDwtIGMoY2xhc3NMYWJlbCkNCg0KZGF0YSA8LSBkYXRhLmZyYW1lKGNsdXN0ZXIgPSBrbWVhbnNfY2x1c3RlciwgbGFiZWwgPSBncm91bmRfdHJ1dGgpDQoNCg0KICBiY3ViZWQgPC0gZnVuY3Rpb24oZGF0YSkgew0KICBuIDwtIG5yb3coZGF0YSkNCiAgdG90YWxfcHJlY2VzaW9uIDwtIDANCiAgdG90YWxfcmVjYWxsIDwtIDANCg0KZm9yIChpIGluIDE6bikgew0KICBjbHVzdGVyIDwtIGRhdGEkY2x1c3RlcltpXQ0KICBsYWJlbCA8LSBkYXRhJGxhYmVsW2ldDQogICAgDQojIE51bWJlciBvZiBvYmplY3RzIGluIHRoZSBzYW1lIGNhdGVnb3J5IGFuZCBjbHVzdGVyDQppbnRlcnNlY3Rpb24gPC0gc3VtKGRhdGEkbGFiZWxbZGF0YSRjbHVzdGVyID09IGNsdXN0ZXJdID09IGxhYmVsKQ0KICAgIA0KIyBOdW1iZXIgb2Ygb2JqZWN0cyB0aGF0IGFyZSBpbiB0aGUgc2FtZSBjbHVzdGVyDQp0b3RhbF9zYW1lX2NsdXN0ZXIgPC0gc3VtKGRhdGEkY2x1c3RlciA9PSBjbHVzdGVyKQ0KICAgIA0KIyBOdW1iZXIgb2Ygb2JqZWN0cyB0aGF0IGhhdmUgdGhlIHNhbWUgY2F0ZWdvcnkNCnRvdGFsX3NhbWVfY2F0ZWdvcnkgPC0gc3VtKGRhdGEkbGFiZWwgPT0gbGFiZWwpDQogICAgDQoNCnRvdGFsX3ByZWNlc2lvbiA8LSB0b3RhbF9wcmVjZXNpb24gKyBpbnRlcnNlY3Rpb24gL3RvdGFsX3NhbWVfY2x1c3Rlcg0KdG90YWxfcmVjYWxsIDwtIHRvdGFsX3JlY2FsbCArIGludGVyc2VjdGlvbiAvIHRvdGFsX3NhbWVfY2F0ZWdvcnkNCiAgfQ0KDQogICMgY29tcHV0ZSBhdmcgcHJlY2lzaW9uIGFuZCByZWNhbGwNCiAgcHJlY2lzaW9uIDwtIHRvdGFsX3ByZWNlc2lvbiAvIG4NCiAgcmVjYWxsIDwtIHRvdGFsX3JlY2FsbCAvIG4NCg0KICByZXR1cm4obGlzdChwcmVjaXNpb24gPSBwcmVjaXNpb24sIHJlY2FsbCA9IHJlY2FsbCkpIH0NCg0KDQojIGNvbXB1dGUgQkN1YmVkIHByZWNpc2lvbiBhbmQgcmVjYWxsDQptZXRyaWNzIDwtIGJjdWJlZChkYXRhKQ0KDQoNCnByZWNpc2lvbiA8LSBtZXRyaWNzJHByZWNpc2lvbg0KcmVjYWxsIDwtIG1ldHJpY3MkcmVjYWxsDQoNCiMgUHJpbnQgcmVzdWx0cw0KY2F0KCJCQ3ViZWQgUHJlY2lzaW9uIGlzOiIsIHByZWNpc2lvbiwgIlxuIikNCmNhdCgiQkN1YmVkIFJlY2FsbCBpczoiLCByZWNhbGwsICJcbiIpDQpgYGANCg0Kd2UgY2FuIGNvbmNsdWRlIGZyb20gdGhlIGdyYXBoIGFuZCB0aGUgcmVzdWx0cyB0aGF0IHRoZSBrPTIgaXMgdGhlIG9wdGltYWwgaywgc2luY2UgdGhlcmUgaXMgbm8gb3ZlcmxhcHBpbmcgYmV0d2VlbiB0aGUgdHdvIGNsdXN0ZXJzLCB0aGUgZGF0YSBpbiBhIGNsdXN0ZXIgYXJlIGNsb3NlICJzaW1pbGFyIiB0byBlYWNoIG90aGVyIGFuZCBkaXNzaW1pbGFyIHRvIGRhdGEgaW4gdGhlIG90aGVyIGNsdXN0ZXIuIEFsc28sIHRoZSByZWNhbGwgaXMgcmVsYXRpdmVseSBoaWdoICgwLjcxKSBhbmQgaXMgdGhlIGhpZ2hlc3QgYW1vbmcgdGhlIGsncyBjaG9zZW4sIHRoZSBQcmVjaXNpb24gaXMgbG93ICgwLjI4KSB3aGljaCBjb3VsZCBiZSBkdW8gdG8gcHJlc2VuY2Ugb2Ygb3V0bGllcnMgb3Igc2Vuc2l0aXZpdHkgdG8gSW5pdGlhbCBDZW50cm9pZC4NCldlIGNhbiBhbHNvIG5vdGUgdGhhdCB0aGUgV1NTIGlzIDI2LjglLCB3aGljaCBpcyBhbiBleGNlbGxlbnQgcGVyY2VudGFnZSBpbmRpY2F0aW5nIGEgZ29vZCBjb21wYWN0bmVzcyBvZiBjbHVzdGVycywgYW5kIHRoYXQgb2JqZWN0cyBpbiBhIGNsdXN0ZXIgYXJlIHNpbWlsYXIgdG8gb25lIGFub3RoZXIuDQpMYXN0bHksIHRoZSBhdmVyYWdlIHNpbGhvdWV0dGUgd2lkdGggaXMgMC4zNCB3aGljaCBpcyBjb25zaWRlcmVkIGhpZ2ggcmVmbGVjdGluZyBoaWdoIGludHJhLWNsdXN0ZXIgc2ltaWxhcml0eS4NCg0KDQojIyMjIGs9Mw0KDQpgYGB7cn0NCg0KI1VzZSBzZWVkIHRvIGd1YXJhbnRlZSByZXBsaWNhYmlsaXR5IG9mIHJhbmRvbSBwcm9jZXNzZXMNCnNldC5zZWVkKDg5NTMpDQoNCiMgcnVuIGstbWVhbnMgY2x1c3RlcmluZyB0byBmaW5kIDMgY2x1c3RlcnMNCmttZWFucy5yZXN1bHQgPC0ga21lYW5zKGRhdGFzZXQzLCAzKQ0KDQojIHByaW50IHRoZSBjbHVzdGVybmcgcmVzdWx0DQprbWVhbnMucmVzdWx0DQoNCiMgdmlzdWFsaXplIGNsdXN0ZXJpbmcNCmZ2aXpfY2x1c3RlcihrbWVhbnMucmVzdWx0LCBkYXRhID0gZGF0YXNldDMpDQoNCiNhdmVyYWdlIHNpbGhvdWV0dGUgZm9yIGVhY2ggY2x1c3RlcnMNCmF2Z19zaWwgPC0gc2lsaG91ZXR0ZShrbWVhbnMucmVzdWx0JGNsdXN0ZXIsZGlzdChkYXRhc2V0MykpIA0KZnZpel9zaWxob3VldHRlKGF2Z19zaWwpDQoNCiNXaXRoaW4tY2x1c3RlciBzdW0gb2Ygc3F1YXJlcyB3c3MgDQp3c3MgPC0ga21lYW5zLnJlc3VsdCR0b3Qud2l0aGluc3MNCnByaW50KHdzcykNCg0KI0JDdWJlZCANCmttZWFuc19jbHVzdGVyIDwtIGMoa21lYW5zLnJlc3VsdCRjbHVzdGVyKQ0KDQpncm91bmRfdHJ1dGggPC0gYyhjbGFzc0xhYmVsKQ0KDQpkYXRhIDwtIGRhdGEuZnJhbWUoY2x1c3RlciA9IGttZWFuc19jbHVzdGVyLCBsYWJlbCA9IGdyb3VuZF90cnV0aCkNCg0KICBiY3ViZWQgPC0gZnVuY3Rpb24oZGF0YSkgew0KICBuIDwtIG5yb3coZGF0YSkNCiAgdG90YWxfcHJlY2VzaW9uIDwtIDANCiAgdG90YWxfcmVjYWxsIDwtIDANCg0KICBmb3IgKGkgaW4gMTpuKSB7DQogICAgY2x1c3RlciA8LSBkYXRhJGNsdXN0ZXJbaV0NCiAgICBsYWJlbCA8LSBkYXRhJGxhYmVsW2ldDQogICAgDQojIE51bWJlciBvZiBvYmplY3RzIGluIHRoZSBzYW1lIGNhdGVnb3J5IGFuZCBjbHVzdGVyDQppbnRlcnNlY3Rpb24gPC0gc3VtKGRhdGEkbGFiZWxbZGF0YSRjbHVzdGVyID09IGNsdXN0ZXJdID09IGxhYmVsKQ0KICAgIA0KIyBOdW1iZXIgb2Ygb2JqZWN0cyB0aGF0IGFyZSBpbiB0aGUgc2FtZSBjbHVzdGVyDQp0b3RhbF9zYW1lX2NsdXN0ZXIgPC0gc3VtKGRhdGEkY2x1c3RlciA9PSBjbHVzdGVyKQ0KICAgIA0KIyBOdW1iZXIgb2Ygb2JqZWN0cyB0aGF0IGhhdmUgdGhlIHNhbWUgY2F0ZWdvcnkNCnRvdGFsX3NhbWVfY2F0ZWdvcnkgPC0gc3VtKGRhdGEkbGFiZWwgPT0gbGFiZWwpDQogICAgDQoNCnRvdGFsX3ByZWNlc2lvbiA8LSB0b3RhbF9wcmVjZXNpb24gKyBpbnRlcnNlY3Rpb24gL3RvdGFsX3NhbWVfY2x1c3Rlcg0KdG90YWxfcmVjYWxsIDwtIHRvdGFsX3JlY2FsbCArIGludGVyc2VjdGlvbiAvIHRvdGFsX3NhbWVfY2F0ZWdvcnkNCiAgfQ0KDQogICMgY29tcHV0ZSBhdmcgcHJlY2lzaW9uIGFuZCByZWNhbGwNCiAgcHJlY2lzaW9uIDwtIHRvdGFsX3ByZWNlc2lvbiAvIG4NCiAgcmVjYWxsIDwtIHRvdGFsX3JlY2FsbCAvIG4NCg0KICByZXR1cm4obGlzdChwcmVjaXNpb24gPSBwcmVjaXNpb24sIHJlY2FsbCA9IHJlY2FsbCkpDQp9DQoNCiMgY29tcHV0ZSBCQ3ViZWQgcHJlY2lzaW9uIGFuZCByZWNhbGwNCm1ldHJpY3MgPC0gYmN1YmVkKGRhdGEpDQoNCg0KcHJlY2lzaW9uIDwtIG1ldHJpY3MkcHJlY2lzaW9uDQpyZWNhbGwgPC0gbWV0cmljcyRyZWNhbGwNCg0KIyBQcmludCByZXN1bHRzDQpjYXQoIkJDdWJlZCBQcmVjaXNpb24gaXM6IiwgcHJlY2lzaW9uLCAiXG4iKQ0KY2F0KCJCQ3ViZWQgUmVjYWxsIGlzOiIsIHJlY2FsbCwgIlxuIikNCmBgYA0KDQp3ZSBjYW4gY29uY2x1ZGUgZnJvbSB0aGUgZ3JhcGggYW5kIHRoZSByZXN1bHRzIHdoZXJlIGs9MyBpcyB0aGF0IHRoZSBwZXJmb3JtYW5jZSBpcyBnb29kIGJ1dCB3b3JzZSB0aGFuIGs9MiwgYmVjYXVzZSB0aGVyZSBpcyBvdmVybGFwcGluZyBiZXR3ZWVuIGNsdXN0ZXJzLiBJbiBhZGRpdGlvbiwgdGhlIHJlY2FsbCBpcyByZWxhdGl2ZWx5IGhpZ2ggKDAuNjgpLCBIb3dldmVyLCB0aGUgUHJlY2lzaW9uIGlzIGxvdyAoMC4yOCkgd2hpY2ggY291bGQgYmUgZHVvIHRvIHByZXNlbmNlIG9mIG91dGxpZXJzIG9yIHNlbnNpdGl2aXR5IHRvIEluaXRpYWwgQ2VudHJvaWQuDQpXZSBjYW4gYWxzbyBub3RlIHRoYXQgdGhlIFdTUyBpcyAzMiUsIHdoaWNoIGlzIGEgZ29vZCBwZXJjZW50YWdlIGluZGljYXRpbmcgYW4gaW50ZXJtaWRpYXRlIGNvbXBhY3RuZXNzIG9mIGNsdXN0ZXJzLCBhbmQgdGhhdCBvYmplY3RzIGluIGEgY2x1c3RlciBhcmUgdG8gc29tZSBleHRlbnQgc2ltaWxhciB0byBvbmUgYW5vdGhlci4NCkxhc3RseSwgdGhlIGF2ZXJhZ2Ugc2lsaG91ZXR0ZSB3aWR0aCBpcyAwLjMxIHdoaWNoIGlzIGdvb2QgcmVmbGVjdGluZyBhIGdvb2QgaW50cmEtY2x1c3RlciBzaW1pbGFyaXR5Lg0KDQojIyMjIGs9NA0KDQpgYGB7cn0NCiNVc2Ugc2VlZCB0byBndWFyYW50ZWUgcmVwbGljYWJpbGl0eSBvZiByYW5kb20gcHJvY2Vzc2VzDQpzZXQuc2VlZCg4OTUzKQ0KDQojIHJ1biBrLW1lYW5zIGNsdXN0ZXJpbmcgdG8gZmluZCA0IGNsdXN0ZXJzDQprbWVhbnMucmVzdWx0IDwtIGttZWFucyhkYXRhc2V0MywgNCkNCg0KIyBwcmludCB0aGUgY2x1c3Rlcm5nIHJlc3VsdA0Ka21lYW5zLnJlc3VsdA0KDQojIHZpc3VhbGl6ZSBjbHVzdGVyaW5nDQpmdml6X2NsdXN0ZXIoa21lYW5zLnJlc3VsdCwgZGF0YSA9IGRhdGFzZXQzKQ0KDQojYXZlcmFnZSBzaWxob3VldHRlIGZvciBlYWNoIGNsdXN0ZXJzDQphdmdfc2lsIDwtIHNpbGhvdWV0dGUoa21lYW5zLnJlc3VsdCRjbHVzdGVyLGRpc3QoZGF0YXNldDMpKSANCmZ2aXpfc2lsaG91ZXR0ZShhdmdfc2lsKQ0KDQojV2l0aGluLWNsdXN0ZXIgc3VtIG9mIHNxdWFyZXMgd3NzIA0Kd3NzIDwtIGttZWFucy5yZXN1bHQkdG90LndpdGhpbnNzDQpwcmludCh3c3MpDQoNCiNCQ3ViZWQNCmttZWFuc19jbHVzdGVyIDwtIGMoa21lYW5zLnJlc3VsdCRjbHVzdGVyKQ0KDQpncm91bmRfdHJ1dGggPC0gYyhjbGFzc0xhYmVsKQ0KDQpkYXRhIDwtIGRhdGEuZnJhbWUoY2x1c3RlciA9IGttZWFuc19jbHVzdGVyLCBsYWJlbCA9IGdyb3VuZF90cnV0aCkNCg0KDQogIGJjdWJlZCA8LSBmdW5jdGlvbihkYXRhKSB7DQogIG4gPC0gbnJvdyhkYXRhKQ0KICB0b3RhbF9wcmVjZXNpb24gPC0gMA0KICB0b3RhbF9yZWNhbGwgPC0gMA0KDQogIGZvciAoaSBpbiAxOm4pIHsNCiAgICBjbHVzdGVyIDwtIGRhdGEkY2x1c3RlcltpXQ0KICAgIGxhYmVsIDwtIGRhdGEkbGFiZWxbaV0NCiAgICANCiMgTnVtYmVyIG9mIG9iamVjdHMgaW4gdGhlIHNhbWUgY2F0ZWdvcnkgYW5kIGNsdXN0ZXINCmludGVyc2VjdGlvbiA8LSBzdW0oZGF0YSRsYWJlbFtkYXRhJGNsdXN0ZXIgPT0gY2x1c3Rlcl0gPT0gbGFiZWwpDQogICAgDQojIE51bWJlciBvZiBvYmplY3RzIGluIHRoZSBzYW1lICBjbHVzdGVyDQp0b3RhbF9zYW1lX2NsdXN0ZXIgPC0gc3VtKGRhdGEkY2x1c3RlciA9PSBjbHVzdGVyKQ0KICAgIA0KIyBOdW1iZXIgb2Ygb2JqZWN0cyB0aGF0IGhhdmUgdGhlIHNhbWUgY2F0ZWdvcnkNCnRvdGFsX3NhbWVfY2F0ZWdvcnkgPC0gc3VtKGRhdGEkbGFiZWwgPT0gbGFiZWwpDQogICAgDQojIENhbGN1bGF0ZSBwcmVjaXNpb24gYW5kIHJlY2FsbCBmb3IgdGhlIGN1cnJlbnQgaXRlbSBhbmQgYWRkIHRoZW0gdG8gdGhlIHN1bXMNCnRvdGFsX3ByZWNlc2lvbiA8LSB0b3RhbF9wcmVjZXNpb24gKyBpbnRlcnNlY3Rpb24gL3RvdGFsX3NhbWVfY2x1c3Rlcg0KdG90YWxfcmVjYWxsIDwtIHRvdGFsX3JlY2FsbCArIGludGVyc2VjdGlvbiAvIHRvdGFsX3NhbWVfY2F0ZWdvcnkNCiAgfQ0KDQogICMgQ29tcHV0ZSBhdmcgcHJlY2lzaW9uIGFuZCByZWNhbGwNCiAgcHJlY2lzaW9uIDwtIHRvdGFsX3ByZWNlc2lvbiAvIG4NCiAgcmVjYWxsIDwtIHRvdGFsX3JlY2FsbCAvIG4NCg0KICByZXR1cm4obGlzdChwcmVjaXNpb24gPSBwcmVjaXNpb24sIHJlY2FsbCA9IHJlY2FsbCkpDQp9DQoNCiMgY29tcHV0ZSBCQ3ViZWQgcHJlY2lzaW9uIGFuZCByZWNhbGwNCm1ldHJpY3MgPC0gYmN1YmVkKGRhdGEpDQoNCg0KcHJlY2lzaW9uIDwtIG1ldHJpY3MkcHJlY2lzaW9uDQpyZWNhbGwgPC0gbWV0cmljcyRyZWNhbGwNCg0KIyBQcmludCByZXN1bHRzDQpjYXQoIkJDdWJlZCBQcmVjaXNpb24gaXM6IiwgcHJlY2lzaW9uLCAiXG4iKQ0KY2F0KCJCQ3ViZWQgUmVjYWxsIGlzOiIsIHJlY2FsbCwgIlxuIikNCmBgYA0KDQp3ZSBjYW4gY29uY2x1ZGUgZnJvbSB0aGUgZ3JhcGggYW5kIHRoZSByZXN1bHRzIHdoZXJlIGs9NCBpcyB0aGF0IHRoZSBwZXJmb3JtYW5jZSBpcyB3b3JzZSB0aGFuIGs9MiBhbmQgaz0zLCBiZWNhdXNlIHRoZXJlIGlzIGEgbm90aWNlYWJsZSBvdmVybGFwcGluZyBiZXR3ZWVuIGNsdXN0ZXJzLiBBbHNvLCB0aGUgY2x1c2Vycycgc3BhY2UgaXMgcHJldHR5IHdpZGUgd2hpY2ggcmVzdWx0cyBpbiBhIGxhcmdlIGRpc3RhbmNlIGJldHdlZW4gb2JqZWN0cyBpbiB0aGUgc2FtZSBjbHVzdGVyLg0KSW4gYWRkaXRpb24sIHRoZSByZWNhbGwgaXMgcmVsYXRpdmVseSBsb3cgKDAuNDApIHdoaWNoIG1pZ2h0IGJlIGEgcmVzdWx0IG9mIHRoZSBvdmVybGFwcGluZyBhbmQgbGFyZ2UgZGlzdGFuY2VzIGJldHdlZW4gZGF0YSBvYmplY3RzLiBGdXJ0aGVybW9yZSwgdGhlIFByZWNpc2lvbiBpcyBsb3cgKDAuMjgpIHdoaWNoIGNvdWxkIGJlIGR1byB0byBwcmVzZW5jZSBvZiBvdXRsaWVycyBvciBzZW5zaXRpdml0eSB0byBJbml0aWFsIENlbnRyb2lkLg0KV2UgY2FuIGFsc28gbm90ZSB0aGF0IHRoZSBXU1MgaXMgNDAuNyUsIHdoaWNoIGlzIGFuIGFjY2VwdGFibGUgcGVyY2VudGFnZSBpbmRpY2F0aW5nIGEgbG93ZXIgY29tcGFjdG5lc3Mgb2YgY2x1c3RlcnMuDQpMYXN0bHksIHRoZSBhdmVyYWdlIHNpbGhvdWV0dGUgd2lkdGggaXMgMC4yIHdoaWNoIGlzIGxvdyByZWZsZWN0aW5nIGhpZ2ggaW50ZXItY2x1c3RlciBzaW1pbGFyaXR5Lg0KDQoNCg==